Skip to content

Instantly share code, notes, and snippets.

View shrijeet's full-sized avatar

Shrijeet shrijeet

9 followers · 4 following
  • Redwood City, CA
View GitHub Profile
@shrijeet
shrijeet / RenameTable.java
Created October 18, 2012 17:30
A program to rename HBase table (works for 0.92.x)
package hbase.experiments;
import java.io.FileNotFoundException;
import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
@shrijeet
shrijeet / FilterRowsWithGivenColumns.java
Created October 16, 2012 23:46
FilterRowsWithGivenColumns MapReduce program
package hbase.experiments;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
@shrijeet
shrijeet / jt_hung.java
Created October 8, 2012 21:12
JT hung
2012-10-05 22:51:31
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode):
"IPC Client (47) connection to master-hadoop.inw-hercules.rfiserve.net/172.22.24.45:8020 from hdfs" daemon prio=10 tid=0x00007f13b4fa5000 nid=0x48d6 in Object.wait() [0x00007f09c8855000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
@shrijeet
shrijeet / HBaseIncExample.java
Created September 20, 2012 23:50
[bug report] Client shutdown (without explicit flush) does not flush buffered atomic increments
import org.hbase.async.AtomicIncrementRequest;
import org.hbase.async.HBaseClient;
public class HBaseIncExample {
public static void main(String [] args) throws Exception {
HBaseClient client = new HBaseClient("rasalghul");
AtomicIncrementRequest req = new AtomicIncrementRequest("t0", "k1", "f1", "q1");
for (int i = 0; i < 100; i++) {
@shrijeet
shrijeet / master.log
Created August 24, 2012 22:16
Master Log while NPE
This file has been truncated, but you can view the full file.
Wed Aug 22 20:24:33 EDT 2012 Starting master on ine-60.rfiserve.net
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 278528
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
@shrijeet
shrijeet / root.log
Created August 24, 2012 22:09
Root rs log during NPE
This file has been truncated, but you can view the full file.
Wed Aug 22 20:08:25 EDT 2012 Starting regionserver on ine-48.rfiserve.net
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 794624
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
@shrijeet
shrijeet / npe_rs_0.92.2.log
Created August 24, 2012 20:46
Null pointer exception in region server 0.92.2
This file has been truncated, but you can view the full file.
Wed Aug 22 20:15:03 EDT 2012 Starting regionserver on ine-51.rfiserve.net
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 794624
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
@shrijeet
shrijeet / confs.txt
Created August 21, 2012 00:31
Hbase 0.90.3 --> 0.92.2 which configurations to tweak?
* MSLAB - Okay to leave it turned ON 0.92.2 onwards since we have lots of memory and number of regions per region server is not too crazy high (200 regions/per RS)
* Distributed splitting - Want it!
* Memory accounting is different now - Okay to leave block cache's value same as it was in 0.90.0 since its already set to a liberal value (0.45)
* Experimental off-heap cache - Leave it turned off.
* Region max file size - Currently set to 2GB, we should go up. 4GB will cut our regions / per region server to under 100.
@shrijeet
shrijeet / save_yourself_from_am.xml
Created August 17, 2012 23:08
You wish!
<?xml version='1.0' encoding='UTF-8'?><feed xmlns='http://www.w3.org/2005/Atom' xmlns:apps='http://schemas.google.com/apps/2006'>
<title>Mail Filters</title>
<id>tag:mail.google.com,2008:filters:1345244786967</id>
<updated>2012-08-17T23:07:52Z</updated>
<author>
<name>Shrijeet Paliwal</name>
<email>shrijeet.paliwal@gmail.com</email>
</author>
<entry>
<category term='filter'></category>
@shrijeet
shrijeet / jvm_metrics.txt
Created August 15, 2012 18:45
Namenode increase in logWarn
jvm.metrics: hostName=inw-5.rfiserve.net, processName=NameNode, sessionId=, gcCount=37845, gcTimeMillis=6003961, logError=0, logFatal=0, logInfo=128826946, logWarn=4218, maxMemoryM=10666.6875, memHeapCommittedM=7094.8125, memHeapUsedM=5518.711, memNonHeapCommittedM=27.5, memNonHeapUsedM=27.320305, threadsBlocked=0, threadsNew=0, threadsRunnable=11, threadsTerminated=0, threadsTimedWaiting=8, threadsWaiting=44
jvm.metrics: hostName=inw-5.rfiserve.net, processName=NameNode, sessionId=, gcCount=38113, gcTimeMillis=6041969, logError=0, logFatal=0, logInfo=129717263, logWarn=4235, maxMemoryM=10666.6875, memHeapCommittedM=7094.1875, memHeapUsedM=3677.5535, memNonHeapCommittedM=27.5, memNonHeapUsedM=27.320305, threadsBlocked=0, threadsNew=0, threadsRunnable=9, threadsTerminated=0, threadsTimedWaiting=8, threadsWaiting=46
jvm.metrics: hostName=inw-5.rfiserve.net, processName=NameNode, sessionId=, gcCount=38379, gcTimeMillis=6089254, logError=0, logFatal=0, logInfo=130416613, logWarn=4244, maxMemoryM=10666.6875, memHe