Skip to content

Instantly share code, notes, and snippets.

@alienth

alienth/gist:bbdf8f8191584bbbcb00

Created September 9, 2015 22:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save alienth/bbdf8f8191584bbbcb00 to your computer and use it in GitHub Desktop.
Save alienth/bbdf8f8191584bbbcb00 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
2015-09-09 22:18:45,244 DEBUG [main-EventThread] [HBaseClient.process] - Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-09-09 22:18:45,245 DEBUG [main-EventThread] [HBaseClient.getRootRegion] - Finding the -ROOT- or .META. region in ZooKeeper
2015-09-09 22:18:45,248 DEBUG [main-EventThread] [HBaseClient.process] - Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-09-09 22:18:45,261 INFO [main-EventThread] [HBaseClient.handleMetaZnode] - Connecting to .META. region @ 10.8.0.231:60020
2015-09-09 22:18:45,284 DEBUG [main-EventThread] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179] OPEN
2015-09-09 22:18:45,288 DEBUG [main-EventThread] [RegionClient.sendRpc] - RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:18:45,289 DEBUG [main-EventThread] [RegionClient.sendRpc] - RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 45, 117, 105, 100, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:18:45,289 DEBUG [main-EventThread] [HBaseClient.disconnectZK] - Ignore any DEBUG exception from ZooKeeper
2015-09-09 22:18:45,289 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] BOUND: /10.8.0.231:44308
2015-09-09 22:18:45,290 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] CONNECTED: /10.8.0.231:60020
2015-09-09 22:18:45,291 DEBUG [main-EventThread] [HBaseClient.disconnectZK] - ZooKeeper#close completed in 1219682ns
2015-09-09 22:18:45,303 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 78
2015-09-09 22:18:45,303 DEBUG [New I/O worker #1] [RegionClient.sendQueuedRpcs] - Executing RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:18:45,316 DEBUG [New I/O worker #1] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #0, payload=BigEndianHeapChannelBuffer(ridx=12, widx=132, cap=132) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 116, 9, 8, 0, 26, 3, 71, 101, 116, 32, 1, 105, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, 84, 10, 72, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 18, 6, 10, 4, 105, 110, 102, 111, 88, 1]
2015-09-09 22:18:45,316 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 120
2015-09-09 22:18:45,316 DEBUG [New I/O worker #1] [RegionClient.sendQueuedRpcs] - Executing RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 45, 117, 105, 100, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:18:45,316 DEBUG [New I/O worker #1] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #1, payload=BigEndianHeapChannelBuffer(ridx=12, widx=136, cap=136) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 120, 9, 8, 1, 26, 3, 71, 101, 116, 32, 1, 109, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, 88, 10, 76, 116, 115, 100, 98, 45, 117, 105, 100, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 18, 6, 10, 4, 105, 110, 102, 111, 88, 1]
2015-09-09 22:18:45,316 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 124
2015-09-09 22:18:45,319 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=957, cap=957)
2015-09-09 22:18:45,319 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:18:45,325 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=0, response size=481 bytes, 476 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:18:45,328 INFO [New I/O worker #1] [HBaseClient.discoverRegion] - Added client for region RegionInfo(table="tsdb", region_name="tsdb,,1441757821416.042efa48342a894c34943605952e5467.", stop_key=[0, 0, 5, 85, -27, 76, -16, 0, 0, 1, 0, 0, 1, 0, 0, 2, 0, 0, 4, 0, 0, 3, 0, 6, 116]), which was added to the regions cache. Now we know that RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) is hosting 1 region.
2015-09-09 22:18:45,329 DEBUG [New I/O worker #1] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #2, payload=BigEndianHeapChannelBuffer(ridx=12, widx=158, cap=158) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x8E\t\x08\x02\x1A\x03Get \x01\x82\x01\n9\x08\x01\x125tsdb,,1441757821416.042efa48342a894c34943605952e5467.\x12E\nA\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<P\x01"
2015-09-09 22:18:45,329 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 146
2015-09-09 22:18:45,329 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 9849us
2015-09-09 22:18:45,329 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:18:45,330 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=1, response size=476 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 45, 117, 105, 100, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:18:45,331 DEBUG [New I/O worker #1] [HBaseClient.getIP] - Resolved IP of `co-tsdb02.ds.stackexchange.com' to 10.8.0.232 in 753510ns
2015-09-09 22:18:45,331 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06] OPEN
2015-09-09 22:18:45,331 INFO [New I/O worker #1] [HBaseClient.discoverRegion] - Added client for region RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.", stop_key=""), which was added to the regions cache. Now we know that RegionClient@1027738753(chan=null, #pending_rpcs=0, #batched=0, #rpcs_inflight=0) is hosting 1 region.
2015-09-09 22:18:45,331 DEBUG [New I/O worker #1] [RegionClient.sendRpc] - RPC queued: Exists(table="tsdb-uid", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.", stop_key=""))
2015-09-09 22:18:45,332 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 2274us
2015-09-09 22:18:45,332 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=14, cap=14)
2015-09-09 22:18:45,332 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] BOUND: /10.8.0.231:34580
2015-09-09 22:18:45,332 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:18:45,332 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] CONNECTED: /10.8.0.232:60020
2015-09-09 22:18:45,332 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=2, response size=14 bytes, 0 readable bytes left, rpc=Exists(table="tsdb", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="tsdb", region_name="tsdb,,1441757821416.042efa48342a894c34943605952e5467.", stop_key=[0, 0, 5, 85, -27, 76, -16, 0, 0, 1, 0, 0, 1, 0, 0, 2, 0, 0, 4, 0, 0, 3, 0, 6, 116]))
2015-09-09 22:18:45,332 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] WRITTEN_AMOUNT: 78
2015-09-09 22:18:45,332 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 568us
2015-09-09 22:18:45,333 DEBUG [New I/O worker #2] [RegionClient.sendQueuedRpcs] - Executing RPC queued: Exists(table="tsdb-uid", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.", stop_key=""))
2015-09-09 22:18:45,333 DEBUG [New I/O worker #2] [RegionClient.encode] - [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] Sending RPC #0, payload=BigEndianHeapChannelBuffer(ridx=12, widx=162, cap=162) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x92\t\x08\x00\x1A\x03Get \x01\x86\x01\n=\x08\x01\x129tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.\x12E\nA\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<P\x01"
2015-09-09 22:18:45,333 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] WRITTEN_AMOUNT: 150
2015-09-09 22:18:45,333 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=14, cap=14)
2015-09-09 22:18:45,334 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:18:45,335 DEBUG [New I/O worker #2] [RegionClient.decode] - rpcid=0, response size=14 bytes, 0 readable bytes left, rpc=Exists(table="tsdb-uid", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.", stop_key=""))
2015-09-09 22:18:45,335 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 1303us
2015-09-09 22:18:45,387 WARN [main] [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.tsd.HttpSerializer
2015-09-09 22:19:03,576 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])] [RegionClient.encode] - [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] Sending RPC #1, payload=BigEndianHeapChannelBuffer(ridx=12, widx=115, cap=115) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00c\t\x08\x01\x1A\x03Get \x01X\n=\x08\x01\x129tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.\x12\x17\n\x06os.cpu\x12\x0D\n\x02id\x12\x07metrics"
2015-09-09 22:19:03,577 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] WRITTEN_AMOUNT: 103
2015-09-09 22:19:03,577 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=49, cap=49)
2015-09-09 22:19:03,577 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:03,577 DEBUG [New I/O worker #2] [RegionClient.decode] - rpcid=1, response size=49 bytes, 0 readable bytes left, rpc=GetRequest(table="tsdb-uid", key="os.cpu", family="id", qualifiers=["metrics"], attempt=1, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.", stop_key=""))
2015-09-09 22:19:03,578 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 1022us
2015-09-09 22:19:03,578 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])] [RegionClient.encode] - [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] Sending RPC #2, payload=BigEndianHeapChannelBuffer(ridx=12, widx=110, cap=110) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00^\t\x08\x02\x1A\x03Get \x01S\n=\x08\x01\x129tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.\x12\x12\n\x04host\x12\n\n\x02id\x12\x04tagk"
2015-09-09 22:19:03,579 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] WRITTEN_AMOUNT: 98
2015-09-09 22:19:03,579 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=44, cap=44)
2015-09-09 22:19:03,579 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:03,579 DEBUG [New I/O worker #2] [RegionClient.decode] - rpcid=2, response size=44 bytes, 0 readable bytes left, rpc=GetRequest(table="tsdb-uid", key="host", family="id", qualifiers=["tagk"], attempt=1, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.", stop_key=""))
2015-09-09 22:19:03,579 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 438us
2015-09-09 22:19:03,580 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])] [RegionClient.encode] - [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] Sending RPC #3, payload=BigEndianHeapChannelBuffer(ridx=12, widx=118, cap=118) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00f\t\x08\x03\x1A\x03Get \x01[\n=\x08\x01\x129tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.\x12\x1A\n\x0Cny-jharvey08\x12\n\n\x02id\x12\x04tagv"
2015-09-09 22:19:03,580 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] WRITTEN_AMOUNT: 106
2015-09-09 22:19:03,580 DEBUG [New I/O worker #2] [RegionClient.handleUpstream] - handleUpstream [id: 0xa809ac06, /10.8.0.231:34580 => /10.8.0.232:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=52, cap=52)
2015-09-09 22:19:03,580 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:03,580 DEBUG [New I/O worker #2] [RegionClient.decode] - rpcid=3, response size=52 bytes, 0 readable bytes left, rpc=GetRequest(table="tsdb-uid", key="ny-jharvey08", family="id", qualifiers=["tagv"], attempt=1, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1438896630731.88026da01db508529f02836d4154bcab.", stop_key=""))
2015-09-09 22:19:03,581 DEBUG [New I/O worker #2] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 415us
2015-09-09 22:19:03,584 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #3, payload=BigEndianHeapChannelBuffer(ridx=12, widx=80, cap=80) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 9, 8, 3, 26, 3, 71, 101, 116, 32, 1, 53, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, 32, 10, 20, 116, 115, 100, 98, 44, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 44, 58, 18, 6, 10, 4, 105, 110, 102, 111, 88, 1]
2015-09-09 22:19:03,584 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 68
2015-09-09 22:19:03,585 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=607, cap=607)
2015-09-09 22:19:03,585 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:03,585 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=3, response size=607 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:19:03,586 INFO [New I/O worker #1] [HBaseClient.discoverRegion] - Added client for region RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]), which was added to the regions cache. Now we know that RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) is hosting 2 regions.
2015-09-09 22:19:03,588 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 3419us
2015-09-09 22:19:04,449 DEBUG [Hashed wheel timer #2] [RegionClient.periodicFlush] - Periodic flush timer: flushing RPCs for RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=1, #rpcs_inflight=0)
2015-09-09 22:19:04,462 DEBUG [Hashed wheel timer #2] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #4, payload=BigEndianHeapChannelBuffer(ridx=12, widx=175, cap=175) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -97, 12, 8, 4, 26, 6, 77, 117, 116, 97, 116, 101, 32, 1, -112, 1, 10, 82, 8, 1, 18, 78, 116, 115, 100, 98, 44, 0, 0, -43, 85, -52, -80, -16, 0, 0, 3, 0, 2, 22, 0, 0, 8, 0, 0, 15, 0, 0, 14, 0, 0, 73, 44, 49, 52, 51, 57, 53, 49, 52, 49, 53, 53, 54, 55, 54, 46, 57, 100, 102, 51, 49, 50, 97, 57, 102, 48, 100, 50, 54, 53, 102, 52, 99, 51, 48, 98, 99, 99, 49, 53, 55, 48, 49, 57, 50, 48, 97, 97, 46, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 71, 107, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0]
2015-09-09 22:19:04,462 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 163
2015-09-09 22:19:04,465 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=8, cap=8)
2015-09-09 22:19:04,465 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:04,466 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=4, response size=8 bytes, 0 readable bytes left, rpc=AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["Gk?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=2, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))
2015-09-09 22:19:04,467 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 1739us
2015-09-09 22:19:19,729 DEBUG [Hashed wheel timer #2] [RegionClient.periodicFlush] - Periodic flush timer: flushing RPCs for RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=1, #rpcs_inflight=0)
2015-09-09 22:19:19,730 DEBUG [Hashed wheel timer #2] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #5, payload=BigEndianHeapChannelBuffer(ridx=12, widx=175, cap=175) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -97, 12, 8, 5, 26, 6, 77, 117, 116, 97, 116, 101, 32, 1, -112, 1, 10, 82, 8, 1, 18, 78, 116, 115, 100, 98, 44, 0, 0, -43, 85, -52, -80, -16, 0, 0, 3, 0, 2, 22, 0, 0, 8, 0, 0, 15, 0, 0, 14, 0, 0, 73, 44, 49, 52, 51, 57, 53, 49, 52, 49, 53, 53, 54, 55, 54, 46, 57, 100, 102, 51, 49, 50, 97, 57, 102, 48, 100, 50, 54, 53, 102, 52, 99, 51, 48, 98, 99, 99, 49, 53, 55, 48, 49, 57, 50, 48, 97, 97, 46, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 72, 91, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0]
2015-09-09 22:19:19,730 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 163
2015-09-09 22:19:19,731 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=8, cap=8)
2015-09-09 22:19:19,732 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:19,732 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=5, response size=8 bytes, 0 readable bytes left, rpc=AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["H[?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))
2015-09-09 22:19:19,732 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 520us
2015-09-09 22:19:21,609 DEBUG [Hashed wheel timer #2] [RegionClient.periodicFlush] - Periodic flush timer: flushing RPCs for RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=1, #rpcs_inflight=0)
2015-09-09 22:19:21,610 DEBUG [Hashed wheel timer #2] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #6, payload=BigEndianHeapChannelBuffer(ridx=12, widx=175, cap=175) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -97, 12, 8, 6, 26, 6, 77, 117, 116, 97, 116, 101, 32, 1, -112, 1, 10, 82, 8, 1, 18, 78, 116, 115, 100, 98, 44, 0, 0, -43, 85, -52, -80, -16, 0, 0, 3, 0, 2, 22, 0, 0, 8, 0, 0, 15, 0, 0, 14, 0, 0, 73, 44, 49, 52, 51, 57, 53, 49, 52, 49, 53, 53, 54, 55, 54, 46, 57, 100, 102, 51, 49, 50, 97, 57, 102, 48, 100, 50, 54, 53, 102, 52, 99, 51, 48, 98, 99, 99, 49, 53, 55, 48, 49, 57, 50, 48, 97, 97, 46, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 72, -117, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0]
2015-09-09 22:19:21,610 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 163
2015-09-09 22:19:21,611 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=8, cap=8)
2015-09-09 22:19:21,612 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:21,612 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=6, response size=8 bytes, 0 readable bytes left, rpc=AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=[[72, -117, 63, 0, 0, 0]], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))
2015-09-09 22:19:21,612 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 551us
2015-09-09 22:19:22,789 DEBUG [Hashed wheel timer #2] [RegionClient.periodicFlush] - Periodic flush timer: flushing RPCs for RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=1, #rpcs_inflight=0)
2015-09-09 22:19:22,790 DEBUG [Hashed wheel timer #2] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #7, payload=BigEndianHeapChannelBuffer(ridx=12, widx=175, cap=175) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -97, 12, 8, 7, 26, 6, 77, 117, 116, 97, 116, 101, 32, 1, -112, 1, 10, 82, 8, 1, 18, 78, 116, 115, 100, 98, 44, 0, 0, -43, 85, -52, -80, -16, 0, 0, 3, 0, 2, 22, 0, 0, 8, 0, 0, 15, 0, 0, 14, 0, 0, 73, 44, 49, 52, 51, 57, 53, 49, 52, 49, 53, 53, 54, 55, 54, 46, 57, 100, 102, 51, 49, 50, 97, 57, 102, 48, 100, 50, 54, 53, 102, 52, 99, 51, 48, 98, 99, 99, 49, 53, 55, 48, 49, 57, 50, 48, 97, 97, 46, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 72, -101, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0]
2015-09-09 22:19:22,790 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 163
2015-09-09 22:19:22,791 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=8, cap=8)
2015-09-09 22:19:22,792 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:22,792 DEBUG [New I/O worker #1] [RegionClient.decode] - rpcid=7, response size=8 bytes, 0 readable bytes left, rpc=AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=[[72, -101, 63, 0, 0, 0]], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))
2015-09-09 22:19:22,792 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 506us
2015-09-09 22:19:34,229 DEBUG [Hashed wheel timer #2] [RegionClient.periodicFlush] - Periodic flush timer: flushing RPCs for RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=2, #rpcs_inflight=0)
2015-09-09 22:19:34,236 DEBUG [Hashed wheel timer #2] [RegionClient.encode] - [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] Sending RPC #8, payload=BigEndianHeapChannelBuffer(ridx=12, widx=245, cap=245) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -27, 11, 8, 8, 26, 5, 77, 117, 108, 116, 105, 32, 1, -41, 1, 10, -44, 1, 10, 82, 8, 1, 18, 78, 116, 115, 100, 98, 44, 0, 0, -43, 85, -52, -80, -16, 0, 0, 3, 0, 2, 22, 0, 0, 8, 0, 0, 15, 0, 0, 14, 0, 0, 73, 44, 49, 52, 51, 57, 53, 49, 52, 49, 53, 53, 54, 55, 54, 46, 57, 100, 102, 51, 49, 50, 97, 57, 102, 48, 100, 50, 54, 53, 102, 52, 99, 51, 48, 98, 99, 99, 49, 53, 55, 48, 49, 57, 50, 48, 97, 97, 46, 26, 62, 8, 0, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 73, 75, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0, 26, 62, 8, 1, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 73, 91, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0]
2015-09-09 22:19:34,236 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] WRITTEN_AMOUNT: 233
2015-09-09 22:19:34,237 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=10, cap=10)
2015-09-09 22:19:34,238 DEBUG [New I/O worker #1] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:34,240 ERROR [New I/O worker #1] [RegionClient.decode] - Uncaught error during de-serialization of MultiAction(batch=[AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["IK?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])), AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["I[?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))]), rpcid=8
2015-09-09 22:19:34,241 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020] EXCEPTION: org.hbase.async.NonRecoverableException: Uncaught error during de-serialization of MultiAction(batch=[AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["IK?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])), AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["I[?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))]), rpcid=8
2015-09-09 22:19:34,245 ERROR [New I/O worker #1] [RegionClient.exceptionCaught] - Unexpected exception from downstream on [id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020]
org.hbase.async.NonRecoverableException: Uncaught error during de-serialization of MultiAction(batch=[AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["IK?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])), AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["I[?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))]), rpcid=8
at org.hbase.async.RegionClient.decode(RegionClient.java:1394) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1141) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]
at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2802) [asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635) ~[na:1.7.0_79]
at java.util.ArrayList.get(ArrayList.java:411) ~[na:1.7.0_79]
at java.util.Collections$UnmodifiableList.get(Collections.java:1211) ~[na:1.7.0_79]
at org.hbase.async.generated.ClientPB$MultiResponse.getRegionActionResult(ClientPB.java:21170) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.hbase.async.MultiAction.deserialize(MultiAction.java:536) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.hbase.async.RegionClient.decode(RegionClient.java:1384) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
... 20 common frames omitted
2015-09-09 22:19:34,246 INFO [New I/O worker #1] [HBaseClient.removeClientFromCache] - Lost connection with the .META. region
2015-09-09 22:19:34,246 DEBUG [New I/O worker #1] [HBaseClient.removeClientFromCache] - Removed from regions cache: RegionInfo(table="tsdb", region_name="tsdb,,1441757821416.042efa48342a894c34943605952e5467.", stop_key=[0, 0, 5, 85, -27, 76, -16, 0, 0, 1, 0, 0, 1, 0, 0, 2, 0, 0, 4, 0, 0, 3, 0, 6, 116])
2015-09-09 22:19:34,246 DEBUG [New I/O worker #1] [HBaseClient.removeClientFromCache] - Association removed: RegionInfo(table="tsdb", region_name="tsdb,,1441757821416.042efa48342a894c34943605952e5467.", stop_key=[0, 0, 5, 85, -27, 76, -16, 0, 0, 1, 0, 0, 1, 0, 0, 2, 0, 0, 4, 0, 0, 3, 0, 6, 116]) -> RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0)
2015-09-09 22:19:34,246 DEBUG [New I/O worker #1] [HBaseClient.removeClientFromCache] - Removed from regions cache: RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])
2015-09-09 22:19:34,246 DEBUG [New I/O worker #1] [HBaseClient.removeClientFromCache] - Association removed: RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]) -> RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0)
2015-09-09 22:19:34,246 DEBUG [New I/O worker #1] [HBaseClient.removeClientFromCache] - Removed from IP cache: 10.8.0.231:60020 -> RegionClient@1660923468(chan=[id: 0xf166f179, /10.8.0.231:44308 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0)
2015-09-09 22:19:34,247 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 :> /10.8.0.231:60020] DISCONNECTED
2015-09-09 22:19:34,247 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 :> /10.8.0.231:60020] UNBOUND
2015-09-09 22:19:34,247 DEBUG [New I/O worker #1] [RegionClient.handleUpstream] - handleUpstream [id: 0xf166f179, /10.8.0.231:44308 :> /10.8.0.231:60020] CLOSED
2015-09-09 22:19:34,276 INFO [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])] [HBaseClient.getDeferredRoot] - Need to find the .META. region
2015-09-09 22:19:34,278 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [HBaseClient.process] - Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-09-09 22:19:34,278 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [HBaseClient.getRootRegion] - Finding the -ROOT- or .META. region in ZooKeeper
2015-09-09 22:19:34,279 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [HBaseClient.process] - Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-09-09 22:19:34,280 INFO [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [HBaseClient.handleMetaZnode] - Connecting to .META. region @ 10.8.0.231:60020
2015-09-09 22:19:34,280 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce] OPEN
2015-09-09 22:19:34,280 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [RegionClient.sendRpc] - RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:19:34,282 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [HBaseClient.disconnectZK] - Ignore any DEBUG exception from ZooKeeper
2015-09-09 22:19:34,283 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] BOUND: /10.8.0.231:51801
2015-09-09 22:19:34,283 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] CONNECTED: /10.8.0.231:60020
2015-09-09 22:19:34,283 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] WRITTEN_AMOUNT: 78
2015-09-09 22:19:34,283 DEBUG [New I/O worker #3] [RegionClient.sendQueuedRpcs] - Executing RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:19:34,283 DEBUG [Old I/O server worker (parentId: 2079332774, [id: 0x7bf019a6, /0.0.0.0:4242])-EventThread] [HBaseClient.disconnectZK] - ZooKeeper#close completed in 1308784ns
2015-09-09 22:19:34,284 DEBUG [New I/O worker #3] [RegionClient.encode] - [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] Sending RPC #0, payload=BigEndianHeapChannelBuffer(ridx=12, widx=80, cap=80) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 9, 8, 0, 26, 3, 71, 101, 116, 32, 1, 53, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, 32, 10, 20, 116, 115, 100, 98, 44, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 44, 58, 18, 6, 10, 4, 105, 110, 102, 111, 88, 1]
2015-09-09 22:19:34,284 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] WRITTEN_AMOUNT: 68
2015-09-09 22:19:34,284 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=607, cap=607)
2015-09-09 22:19:34,284 DEBUG [New I/O worker #3] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:34,284 DEBUG [New I/O worker #3] [RegionClient.decode] - rpcid=0, response size=607 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0)
2015-09-09 22:19:34,285 INFO [New I/O worker #3] [HBaseClient.discoverRegion] - Added client for region RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]), which was added to the regions cache. Now we know that RegionClient@661161821(chan=[id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) is hosting 1 region.
2015-09-09 22:19:34,285 DEBUG [New I/O worker #3] [RegionClient.decode] - ------------------<< LEAVING DECODE <<------------------ time elapsed: 833us
2015-09-09 22:19:35,429 DEBUG [Hashed wheel timer #2] [RegionClient.periodicFlush] - Periodic flush timer: flushing RPCs for RegionClient@661161821(chan=[id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020], #pending_rpcs=0, #batched=2, #rpcs_inflight=0)
2015-09-09 22:19:35,430 DEBUG [Hashed wheel timer #2] [RegionClient.encode] - [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] Sending RPC #1, payload=BigEndianHeapChannelBuffer(ridx=12, widx=245, cap=245) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -27, 11, 8, 1, 26, 5, 77, 117, 108, 116, 105, 32, 1, -41, 1, 10, -44, 1, 10, 82, 8, 1, 18, 78, 116, 115, 100, 98, 44, 0, 0, -43, 85, -52, -80, -16, 0, 0, 3, 0, 2, 22, 0, 0, 8, 0, 0, 15, 0, 0, 14, 0, 0, 73, 44, 49, 52, 51, 57, 53, 49, 52, 49, 53, 53, 54, 55, 54, 46, 57, 100, 102, 51, 49, 50, 97, 57, 102, 48, 100, 50, 54, 53, 102, 52, 99, 51, 48, 98, 99, 99, 49, 53, 55, 48, 49, 57, 50, 48, 97, 97, 46, 26, 62, 8, 0, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 73, 91, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0, 26, 62, 8, 1, 18, 58, 10, 13, 0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95, 16, 0, 26, 28, 10, 1, 116, 18, 23, 10, 3, 5, 0, 0, 18, 6, 73, 107, 63, 0, 0, 0, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 42, 9, 10, 4, 95, 114, 114, 95, 18, 1, 0]
2015-09-09 22:19:35,430 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] WRITTEN_AMOUNT: 233
2015-09-09 22:19:35,432 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=10, cap=10)
2015-09-09 22:19:35,432 DEBUG [New I/O worker #3] [RegionClient.decode] - ------------------>> ENTERING DECODE >>------------------
2015-09-09 22:19:35,433 ERROR [New I/O worker #3] [RegionClient.decode] - Uncaught error during de-serialization of MultiAction(batch=[AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["I[?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=3, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])), AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["Ik?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))]), rpcid=1
2015-09-09 22:19:35,433 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020] EXCEPTION: org.hbase.async.NonRecoverableException: Uncaught error during de-serialization of MultiAction(batch=[AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["I[?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=3, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])), AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["Ik?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))]), rpcid=1
2015-09-09 22:19:35,434 ERROR [New I/O worker #3] [RegionClient.exceptionCaught] - Unexpected exception from downstream on [id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020]
org.hbase.async.NonRecoverableException: Uncaught error during de-serialization of MultiAction(batch=[AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["I[?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=3, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])), AppendRequest(table="tsdb", key=[0, 0, -38, 85, -16, -85, -32, 0, 0, 3, 0, 24, -95], family="t", qualifiers=[[5, 0, 0]], values=["Ik?\x00\x00\x00"], timestamp=9223372036854775807, lockid=-1, durable=true, return_result=false, bufferable=true, attempt=1, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]))]), rpcid=1
at org.hbase.async.RegionClient.decode(RegionClient.java:1394) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]
at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1141) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]
at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:2802) [asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635) ~[na:1.7.0_79]
at java.util.ArrayList.get(ArrayList.java:411) ~[na:1.7.0_79]
at java.util.Collections$UnmodifiableList.get(Collections.java:1211) ~[na:1.7.0_79]
at org.hbase.async.generated.ClientPB$MultiResponse.getRegionActionResult(ClientPB.java:21170) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.hbase.async.MultiAction.deserialize(MultiAction.java:536) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
at org.hbase.async.RegionClient.decode(RegionClient.java:1384) ~[asynchbase-1.7.0-20150517.200244-1.jar:na]
... 20 common frames omitted
2015-09-09 22:19:35,434 INFO [New I/O worker #3] [HBaseClient.removeClientFromCache] - Lost connection with the .META. region
2015-09-09 22:19:35,434 DEBUG [New I/O worker #3] [HBaseClient.removeClientFromCache] - Removed from regions cache: RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102])
2015-09-09 22:19:35,434 DEBUG [New I/O worker #3] [HBaseClient.removeClientFromCache] - Association removed: RegionInfo(table="tsdb", region_name="tsdb,\x00\x00\xD5U\xCC\xB0\xF0\x00\x00\x03\x00\x02\x16\x00\x00\x08\x00\x00\x0F\x00\x00\x0E\x00\x00I,1439514155676.9df312a9f0d265f4c30bcc15701920aa.", stop_key=[0, 0, -29, 85, -58, -90, 16, 0, 0, 3, 0, 3, 58, 0, 0, 8, 0, 0, 113, 0, 0, 14, 0, 0, 102]) -> RegionClient@661161821(chan=[id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0)
2015-09-09 22:19:35,434 DEBUG [New I/O worker #3] [HBaseClient.removeClientFromCache] - Removed from IP cache: 10.8.0.231:60020 -> RegionClient@661161821(chan=[id: 0xc36c17ce, /10.8.0.231:51801 => /10.8.0.231:60020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0)
2015-09-09 22:19:35,435 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 :> /10.8.0.231:60020] DISCONNECTED
2015-09-09 22:19:35,435 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 :> /10.8.0.231:60020] UNBOUND
2015-09-09 22:19:35,435 DEBUG [New I/O worker #3] [RegionClient.handleUpstream] - handleUpstream [id: 0xc36c17ce, /10.8.0.231:51801 :> /10.8.0.231:60020] CLOSED
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment