Skip to content

Instantly share code, notes, and snippets.

@pfischermx

pfischermx/output2 Secret

Created December 16, 2015 17:46
Show Gist options
  • Save pfischermx/5c3c753db3bdb09efa76 to your computer and use it in GitHub Desktop.
Save pfischermx/5c3c753db3bdb09efa76 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
output2
This file has been truncated, but you can view the full file.
2015-12-16 17:31:50,780 INFO [main] TSDMain: Starting.
2015-12-16 17:31:50,796 INFO [main] TSDMain: net.opentsdb.tools 2.2.0RC3 built at revision (MODIFIED)
2015-12-16 17:31:50,797 INFO [main] TSDMain: Built on 2015/11/11 21:03:33 +0000 by root@centos.localhost:/root/rpmbuild/BUILD/opentsdb-2.2.0RC3
2015-12-16 17:31:50,850 INFO [main] Config: Successfully loaded configuration file: /home/pfs/opentsdb/share/opentsdb/etc/opentsdb/opentsdb.conf
2015-12-16 17:31:50,914 DEBUG [main] SelectorUtil: Using select timeout of 500
2015-12-16 17:31:50,915 DEBUG [main] SelectorUtil: Epoll-bug workaround enabled = false
2015-12-16 17:31:51,760 INFO [main] Config: Successfully loaded configuration file: /home/pfs/opentsdb/share/opentsdb/etc/opentsdb/opentsdb.conf
2015-12-16 17:31:52,028 DEBUG [main] TSDB: TSD Configuration:
File [/home/pfs/opentsdb/share/opentsdb/etc/opentsdb/opentsdb.conf]
Key [tsd.storage.compaction.max_concurrent_flushes] Value [10000]
Key [tsd.storage.flush_interval] Value [1000]
Key [tsd.network.tcp_no_delay] Value [true]
Key [tsd.mode] Value [rw]
Key [tsd.core.preload_uid_cache.max_entries] Value [300000]
Key [tsd.http.staticroot] Value [/home/pfs/opentsdb/share/opentsdb/static]
Key [tsd.storage.hbase.zk_quorum] Value [a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512]
Key [tsd.network.port] Value [9992]
Key [tsd.storage.fix_duplicates] Value [false]
Key [tsd.http.request.cors_domains] Value []
Key [tsd.core.meta.enable_tsuid_tracking] Value [false]
Key [tsd.http.request.cors_headers] Value [Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since]
Key [tsd.storage.hbase.meta_table] Value [perf_namespace:test_opentsdb_meta]
Key [tsd.search.enable] Value [false]
Key [tsd.core.tree.enable_processing] Value [false]
Key [tsd.http.cachedir] Value [/tmp/opentsdb]
Key [tsd.http.request.max_chunk] Value [4096]
Key [tsd.http.show_stack_trace] Value [true]
Key [tsd.query.allow_simultaneous_duplicates] Value [false]
Key [tsd.storage.compaction.flush_speed] Value [2]
Key [tsd.core.uid.random_metrics] Value [false]
Key [tsd.core.meta.enable_realtime_uid] Value [false]
Key [tsd.core.preload_uid_cache] Value [false]
Key [tsd.query.filter.expansion_limit] Value [4096]
Key [tsd.storage.hbase.uid_table] Value [perf_namespace:test_opentsdb_uid]
Key [tsd.storage.repair_appends] Value [false]
Key [tsd.storage.hbase.zk_basedir] Value [/hbase/reluxred-hb1]
Key [tsd.storage.compaction.flush_interval] Value [10]
Key [tsd.query.skip_unresolved_tagvs] Value [false]
Key [hbase.kerberos.regionserver.principal] Value [hbase/_HOST@grid.bar.com]
Key [tsd.no_diediedie] Value [false]
Key [tsd.core.meta.enable_realtime_ts] Value [false]
Key [tsd.http.query.allow_delete] Value [false]
Key [tsd.network.bind] Value [0.0.0.0]
Key [tsd.network.worker_threads] Value []
Key [tsd.core.socket.timeout] Value [0]
Key [hbase.security.authentication] Value [kerberos]
Key [tsd.core.meta.enable_tsuid_incrementing] Value [false]
Key [tsd.core.auto_create_tagvs] Value [true]
Key [tsd.network.async_io] Value [true]
Key [tsd.rtpublisher.plugin] Value []
Key [tsd.search.plugin] Value []
Key [tsd.core.storage_exception_handler.enable] Value [false]
Key [tsd.stats.canonical] Value [false]
Key [tsd.storage.hbase.prefetch_meta] Value [true]
Key [tsd.storage.enable_appends] Value [false]
Key [tsd.core.auto_create_metrics] Value [true]
Key [tsd.storage.enable_compaction] Value [true]
Key [tsd.core.auto_create_tagks] Value [true]
Key [tsd.storage.compaction.min_flush_threshold] Value [100]
Key [tsd.storage.hbase.data_table] Value [perf_namespace:test_opentsdb]
Key [tsd.http.request.enable_chunked] Value [false]
Key [tsd.core.plugin_path] Value [/home/pfs/opentsdb/share/opentsdb/plugins]
Key [hbase.security.auth.enable] Value [true]
Key [tsd.query.timeout] Value [0]
Key [tsd.rtpublisher.enable] Value [false]
Key [hbase.sasl.clientconfig] Value [Client]
Key [tsd.storage.hbase.tree_table] Value [perf_namespace:test_opentsdb_tree]
Key [tsd.network.reuse_address] Value [true]
Key [tsd.network.keep_alive] Value [true]
2015-12-16 17:31:52,035 DEBUG [main] PluginLoader: No JAR files found in path: /home/pfs/opentsdb/share/opentsdb/plugins
2015-12-16 17:31:52,049 WARN [main] PluginLoader: Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter
2015-12-16 17:31:52,049 INFO [main] TSDB: Pre-fetching meta data for all tables
2015-12-16 17:31:52,152 INFO [main] ZooKeeper: Client environment:zookeeper.version=3.4.5-1461890, built on 03/27/2013 23:04 GMT
2015-12-16 17:31:52,153 INFO [main] ZooKeeper: Client environment:java.version=1.7.0_17
2015-12-16 17:31:52,153 INFO [main] ZooKeeper: Client environment:java.vendor=Oracle Corporation
2015-12-16 17:31:52,153 INFO [main] ZooKeeper: Client environment:java.home=/home/y/share/gridjdk-1.7.0_17/jre
2015-12-16 17:31:52,154 INFO [main] ZooKeeper: Client environment:java.class.path=/home/pfs/opentsdb/share/opentsdb/*.jar:/home/pfs/opentsdb/share/opentsdb:/home/pfs/opentsdb/share/opentsdb/bin:/home/pfs/opentsdb/share/opentsdb/lib/async-1.4.0.jar:/home/pfs/opentsdb/share/opentsdb/lib/asynchbase-1.7.1-20151004.015637-1.jar:/home/pfs/opentsdb/share/opentsdb/lib/commons-math3-3.4.1.jar:/home/pfs/opentsdb/share/opentsdb/lib/guava-18.0.jar:/home/pfs/opentsdb/share/opentsdb/lib/jackson-annotations-2.4.3.jar:/home/pfs/opentsdb/share/opentsdb/lib/jackson-core-2.4.3.jar:/home/pfs/opentsdb/share/opentsdb/lib/jackson-databind-2.4.3.jar:/home/pfs/opentsdb/share/opentsdb/lib/log4j-over-slf4j-1.7.7.jar:/home/pfs/opentsdb/share/opentsdb/lib/logback-classic-1.0.13.jar:/home/pfs/opentsdb/share/opentsdb/lib/logback-core-1.0.13.jar:/home/pfs/opentsdb/share/opentsdb/lib/netty-3.9.4.Final.jar:/home/pfs/opentsdb/share/opentsdb/lib/protobuf-java-2.5.0.jar:/home/pfs/opentsdb/share/opentsdb/lib/slf4j-api-1.7.7.jar:/home/pfs/opentsdb/share/opentsdb/lib/tsdb-2.2.0RC3.jar:/home/pfs/opentsdb/share/opentsdb/etc/opentsdb:/home/y/lib/jars/zookeeper_client.jar
2015-12-16 17:31:52,154 INFO [main] ZooKeeper: Client environment:java.library.path=/java/packages/lib/i386:/lib:/lib
2015-12-16 17:31:52,154 INFO [main] ZooKeeper: Client environment:java.io.tmpdir=/tmp
2015-12-16 17:31:52,155 INFO [main] ZooKeeper: Client environment:java.compiler=<NA>
2015-12-16 17:31:52,155 INFO [main] ZooKeeper: Client environment:os.name=Linux
2015-12-16 17:31:52,155 INFO [main] ZooKeeper: Client environment:os.arch=i386
2015-12-16 17:31:52,156 INFO [main] ZooKeeper: Client environment:user.name=root
2015-12-16 17:31:52,156 INFO [main] ZooKeeper: Client environment:user.home=/root
2015-12-16 17:31:52,157 INFO [main] ZooKeeper: Client environment:user.dir=/home/pfs/opentsdb/bin
2015-12-16 17:31:52,159 INFO [main] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:52,180 DEBUG [main] ClientCnxn: zookeeper.disableAutoWatchReset is false
2015-12-16 17:31:52,258 INFO [main] HBaseClient: Need to find the -ROOT- region
2015-12-16 17:31:52,291 DEBUG [main-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: JAAS loginContext is: Client
2015-12-16 17:31:53,306 INFO [main-SendThread(a.a.a.a:50512)] Login: successfully logged in.
2015-12-16 17:31:53,308 INFO [Thread-1] Login: TGT refresh thread started.
2015-12-16 17:31:53,320 DEBUG [Thread-1] Login: Found tgt Ticket (hex) =
2015-12-16 17:31:53,321 INFO [Thread-1] Login: TGT valid starting at: Wed Dec 16 17:31:53 UTC 2015
2015-12-16 17:31:53,321 INFO [Thread-1] Login: TGT expires: Thu Dec 17 17:31:53 UTC 2015
2015-12-16 17:31:53,321 INFO [Thread-1] Login: TGT refresh sleeping until: Thu Dec 17 13:12:49 UTC 2015
2015-12-16 17:31:53,335 INFO [main-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:53,335 DEBUG [main-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:31:53,388 INFO [main-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:53,394 INFO [main-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:31:53,397 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:31:53,405 INFO [main-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bda3, negotiated timeout = 5000
2015-12-16 17:31:53,407 DEBUG [main-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:53,407 DEBUG [main-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:53,409 DEBUG [main-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:53,409 DEBUG [main-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:31:53,417 DEBUG [main-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:53,725 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,727 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,728 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,729 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,730 DEBUG [main-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:53,743 DEBUG [main-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:53,747 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,754 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,756 DEBUG [main-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:53,757 DEBUG [main-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:53,757 DEBUG [main-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bda3
2015-12-16 17:31:53,758 DEBUG [main-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bda3
2015-12-16 17:31:53,760 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,762 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,768 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:53,769 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,771 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:53,772 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:53,776 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda3, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265404431,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:31:53,782 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda3, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265404431,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:31:53,783 DEBUG [main-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda3, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265404432,0 request:: null response:: null
2015-12-16 17:31:53,784 DEBUG [main-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bda3
2015-12-16 17:31:53,784 INFO [main-EventThread] ZooKeeper: Session: 0x151705ac5a7bda3 closed
2015-12-16 17:31:53,785 DEBUG [main-EventThread] HBaseClient: ZooKeeper#close completed in 27336702ns
2015-12-16 17:31:53,785 WARN [main-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:53,785 INFO [main-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:53,795 INFO [main-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:53,795 DEBUG [main-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:31:53,803 INFO [main-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:53,804 INFO [main-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:31:53,805 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:31:53,808 INFO [main-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b55, negotiated timeout = 5000
2015-12-16 17:31:53,809 DEBUG [main-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:53,809 DEBUG [main-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:53,809 DEBUG [main-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:53,809 DEBUG [main-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:31:53,811 DEBUG [main-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:53,869 DEBUG [main-EventThread] HBaseClient: Resolved IP of `x.x.x.x' to 10.192.210.33 in 698256ns
2015-12-16 17:31:53,870 INFO [main-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:31:53,973 DEBUG [main-EventThread] HBaseClient: Channel [id: 0xa87c9ccf]'s state changed: [id: 0xa87c9ccf] OPEN
2015-12-16 17:31:53,992 DEBUG [main-EventThread] RegionClient: handleUpstream [id: 0xa87c9ccf] OPEN
2015-12-16 17:31:53,995 DEBUG [main-EventThread] HBaseClient: Channel [id: 0xa87c9ccf]'s state changed: [id: 0xa87c9ccf] CONNECT: /10.192.210.33:50511
2015-12-16 17:31:54,004 DEBUG [main-EventThread] RegionClient: RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0, timeout=-1, hasTimedout=false)
2015-12-16 17:31:54,009 DEBUG [AsyncHBase I/O Worker #1] HBaseClient: Channel [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511]'s state changed: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] BOUND: /4.4.4.4:41938
2015-12-16 17:31:54,009 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] BOUND: /4.4.4.4:41938
2015-12-16 17:31:54,010 DEBUG [AsyncHBase I/O Worker #1] HBaseClient: Channel [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511]'s state changed: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] CONNECTED: /10.192.210.33:50511
2015-12-16 17:31:54,010 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] CONNECTED: /10.192.210.33:50511
2015-12-16 17:31:54,017 DEBUG [AsyncHBase I/O Worker #1] Login: Constructing login context with context: Client
2015-12-16 17:31:54,020 DEBUG [main-EventThread] Deferred: callback=retry RPC@12949487 returned Deferred@7984521(state=PENDING, result=null, callback=type getClosestRowBefore response -> locateRegion in META -> release .META. lookup permit -> passthrough -> retry RPC -> (continuation of Deferred@11684859 after retry RPC@12949487), errback=passthrough -> passthrough -> release .META. lookup permit -> locateRegion errback -> retry RPC -> (continuation of Deferred@11684859 after retry RPC@12949487)), so the following Deferred is getting paused: Deferred@11684859(state=PAUSED, result=Deferred@7984521, callback=retry ("perf_namespace:test_opentsdb", "", "") -> notify DeferredGroup@26902370, errback=passthrough -> notify DeferredGroup@26902370)
2015-12-16 17:31:54,021 DEBUG [main-EventThread] RegionClient: RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0, timeout=-1, hasTimedout=false)
2015-12-16 17:31:54,021 DEBUG [main-EventThread] Deferred: callback=retry RPC@2176831 returned Deferred@11682722(state=PENDING, result=null, callback=type getClosestRowBefore response -> locateRegion in META -> release .META. lookup permit -> passthrough -> retry RPC -> (continuation of Deferred@24539981 after retry RPC@2176831), errback=passthrough -> passthrough -> release .META. lookup permit -> locateRegion errback -> retry RPC -> (continuation of Deferred@24539981 after retry RPC@2176831)), so the following Deferred is getting paused: Deferred@24539981(state=PAUSED, result=Deferred@11682722, callback=retry ("perf_namespace:test_opentsdb_uid", "", "") -> notify DeferredGroup@26902370, errback=passthrough -> notify DeferredGroup@26902370)
2015-12-16 17:31:54,022 DEBUG [main-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:54,022 DEBUG [main-EventThread] ZooKeeper: Closing session: 0x251705aced21b55
2015-12-16 17:31:54,022 DEBUG [main-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b55
2015-12-16 17:31:54,087 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,088 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,088 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:54,089 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,090 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,091 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:54,092 DEBUG [main-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:54,094 DEBUG [main-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:54,100 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,101 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,102 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:54,102 DEBUG [main-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:54,103 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,104 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,105 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:54,106 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,107 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:54,107 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:54,120 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b55, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265404534,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:31:54,122 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b55, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265404534,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:31:54,123 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b55, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265404535,0 request:: null response:: null
2015-12-16 17:31:54,123 DEBUG [main-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b55
2015-12-16 17:31:54,123 DEBUG [main-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b55 : Unable to read additional data from server sessionid 0x251705aced21b55, likely server has closed socket
2015-12-16 17:31:54,124 INFO [main-EventThread] ZooKeeper: Session: 0x251705aced21b55 closed
2015-12-16 17:31:54,124 DEBUG [main-EventThread] HBaseClient: ZooKeeper#close completed in 101841781ns
2015-12-16 17:31:54,124 INFO [main-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:54,125 INFO [main-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:31:54,125 INFO [main-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:54,296 INFO [AsyncHBase I/O Worker #1] Login: Successfully logged in.
2015-12-16 17:31:54,318 DEBUG [AsyncHBase I/O Worker #1] Login: Found tgt Ticket (hex) =
2015-12-16 17:31:54,319 INFO [AsyncHBase I/O Worker #1] Login: TGT valid starting at: Wed Dec 16 17:31:54 UTC 2015
2015-12-16 17:31:54,320 INFO [AsyncHBase I/O Worker #1] Login: TGT expires: Thu Dec 17 17:31:54 UTC 2015
2015-12-16 17:31:54,321 INFO [AsyncHBase I/O Worker #1] Login: Scheduled ticket renewal in 72966334 ms
2015-12-16 17:31:54,322 INFO [AsyncHBase I/O Worker #1] Login: Initialized kerberos login context
2015-12-16 17:31:54,332 INFO [AsyncHBase I/O Worker #1] KerberosClientAuthProvider: Connecting to hbase/x.x.x.x@grid.bar.com
2015-12-16 17:31:54,334 INFO [AsyncHBase I/O Worker #1] KerberosClientAuthProvider: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:54,335 DEBUG [AsyncHBase I/O Worker #1] KerberosClientAuthProvider: Creating sasl client: client=foo@grid.bar.com, service=hbase, serviceHostname=x.x.x.x
2015-12-16 17:31:54,368 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 6
2015-12-16 17:31:54,739 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 578
2015-12-16 17:31:54,740 INFO [AsyncHBase I/O Worker #1] RegionClient: Initialized security helper: org.hbase.async.SecureRpcHelper96@12a5296 for region client: RegionClient@16233423(chan=null, #pending_rpcs=2, #batched=0, #rpcs_inflight=0)
2015-12-16 17:31:54,746 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=112, cap=112)
2015-12-16 17:31:54,746 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:54,748 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 4
2015-12-16 17:31:54,749 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=58, cap=58)
2015-12-16 17:31:54,749 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:54,751 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 54
2015-12-16 17:31:54,752 INFO [AsyncHBase I/O Worker #1] SecureRpcHelper96: SASL client context established. Negotiated QoP: auth on for: RegionClient@16233423(chan=null, #pending_rpcs=2, #batched=0, #rpcs_inflight=0)
2015-12-16 17:31:54,769 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 91
2015-12-16 17:31:54,770 DEBUG [AsyncHBase I/O Worker #1] RegionClient: Executing RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0, timeout=-1, hasTimedout=false)
2015-12-16 17:31:54,807 DEBUG [AsyncHBase I/O Worker #1] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #0, payload=BigEndianHeapChannelBuffer(ridx=12, widx=159, cap=159) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x8F\t\x08\x00\x1A\x03Get \x01\x83\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12n\nbperf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:31:54,807 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 147
2015-12-16 17:31:54,808 DEBUG [AsyncHBase I/O Worker #1] RegionClient: Executing RPC queued: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0, timeout=-1, hasTimedout=false)
2015-12-16 17:31:54,808 DEBUG [AsyncHBase I/O Worker #1] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #1, payload=BigEndianHeapChannelBuffer(ridx=12, widx=163, cap=163) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x93\t\x08\x01\x1A\x03Get \x01\x87\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12r\nfperf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:31:54,808 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 151
2015-12-16 17:31:54,809 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:31:54,809 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:54,814 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=0, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0, timeout=0, hasTimedout=false)
2015-12-16 17:31:54,814 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) seems to be splitting or closing it.
2015-12-16 17:31:54,815 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:31:54,816 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:54,816 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=1, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=0, timeout=0, hasTimedout=false)
2015-12-16 17:31:55,232 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=null)
2015-12-16 17:31:55,233 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:55,238 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:31:55,249 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:55,250 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:31:55,263 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:55,264 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:31:55,265 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:31:55,267 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec235, negotiated timeout = 5000
2015-12-16 17:31:55,268 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:55,268 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:31:55,269 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:55,268 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:55,270 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:55,547 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,548 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,549 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,550 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,550 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:55,552 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:55,553 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,554 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,554 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:55,554 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:55,555 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec235
2015-12-16 17:31:55,555 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,555 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec235
2015-12-16 17:31:55,556 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,556 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,568 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:55,571 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec235, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265404918,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:31:55,572 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec235, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265404918,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:31:55,573 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec235, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265404919,0 request:: null response:: null
2015-12-16 17:31:55,573 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec235
2015-12-16 17:31:55,574 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec235 : Unable to read additional data from server sessionid 0x351705ac89ec235, likely server has closed socket
2015-12-16 17:31:55,574 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec235 closed
2015-12-16 17:31:55,574 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 19587233ns
2015-12-16 17:31:55,574 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:55,575 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:55,588 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:31:55,588 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:55,589 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:31:55,589 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #2, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 2, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:31:55,590 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@21597776 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776)), so the following Deferred is getting paused: Deferred@8916011(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:31:55,590 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:55,590 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:31:55,591 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:31:55,591 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:55,592 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:31:55,593 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:31:55,593 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:55,594 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=2, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:55,595 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:31:55,596 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:31:55,597 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:31:55,608 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,609 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,610 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,611 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec236, negotiated timeout = 5000
2015-12-16 17:31:55,612 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:55,612 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,613 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,614 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,614 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:55,614 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:55,622 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,623 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,624 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,625 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,626 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:55,630 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:55,631 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,632 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,633 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,634 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:55,636 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec236, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265404946,0 request:: null response:: null
2015-12-16 17:31:55,637 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec236
2015-12-16 17:31:55,637 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec236 : Unable to read additional data from server sessionid 0x351705ac89ec236, likely server has closed socket
2015-12-16 17:31:55,637 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec236 closed
2015-12-16 17:31:55,638 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 47489381ns
2015-12-16 17:31:55,638 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:55,638 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:55,639 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:55,639 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:56,209 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=1, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:56,209 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:56,210 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:31:56,211 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:56,212 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:31:56,213 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:56,214 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:31:56,214 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:31:56,216 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec237, negotiated timeout = 5000
2015-12-16 17:31:56,217 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:56,217 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:56,217 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:56,217 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:31:56,218 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:56,221 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,221 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,222 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,222 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,223 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:56,224 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:56,225 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,225 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,226 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:56,226 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,226 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:56,226 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec237
2015-12-16 17:31:56,226 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec237
2015-12-16 17:31:56,227 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,227 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,228 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:56,229 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec237, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265405117,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:31:56,230 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec237, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265405117,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:31:56,231 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec237, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265405118,0 request:: null response:: null
2015-12-16 17:31:56,231 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec237
2015-12-16 17:31:56,231 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec237 : Unable to read additional data from server sessionid 0x351705ac89ec237, likely server has closed socket
2015-12-16 17:31:56,232 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec237 closed
2015-12-16 17:31:56,232 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 5608046ns
2015-12-16 17:31:56,232 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:56,232 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:56,233 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:31:56,234 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #3, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 3, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:31:56,234 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:56,234 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@24798246 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246)), so the following Deferred is getting paused: Deferred@4531348(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:31:56,234 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:31:56,234 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:31:56,235 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:56,235 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:31:56,235 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:31:56,236 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:31:56,236 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:56,236 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=3, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:56,236 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:56,237 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:31:56,237 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:31:56,238 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:31:56,239 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,239 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,239 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,240 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,240 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,241 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,241 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,241 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bda4, negotiated timeout = 5000
2015-12-16 17:31:56,242 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:56,242 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,242 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,243 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,243 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:56,243 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:56,245 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,246 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,246 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,247 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,247 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,248 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,248 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,248 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,249 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:56,250 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:56,250 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,251 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,251 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,252 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,252 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,253 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,253 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,254 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:56,255 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda4, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265405127,0 request:: null response:: null
2015-12-16 17:31:56,256 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bda4
2015-12-16 17:31:56,256 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bda4 : Unable to read additional data from server sessionid 0x151705ac5a7bda4, likely server has closed socket
2015-12-16 17:31:56,256 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bda4 closed
2015-12-16 17:31:56,256 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 21005766ns
2015-12-16 17:31:56,256 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:56,257 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:56,257 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:56,257 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:57,048 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:57,049 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:57,051 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:31:57,052 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:57,053 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:31:57,055 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:57,056 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:31:57,057 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:31:57,059 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec238, negotiated timeout = 5000
2015-12-16 17:31:57,060 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:57,061 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:57,060 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:57,062 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:31:57,067 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:57,069 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,070 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,072 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,073 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,074 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:57,076 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:57,078 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,079 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,080 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:57,080 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:57,080 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec238
2015-12-16 17:31:57,081 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec238
2015-12-16 17:31:57,082 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,084 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,085 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,086 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,087 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:57,088 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,090 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec238, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265405453,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:31:57,093 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec238, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265405453,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:31:57,094 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec238, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265405454,0 request:: null response:: null
2015-12-16 17:31:57,095 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec238
2015-12-16 17:31:57,095 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec238 closed
2015-12-16 17:31:57,095 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 14936697ns
2015-12-16 17:31:57,096 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:57,096 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:57,102 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:31:57,123 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #4, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 4, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:31:57,123 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:57,124 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@31521182 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182)), so the following Deferred is getting paused: Deferred@19136330(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:31:57,124 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:31:57,124 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:31:57,124 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:57,125 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:31:57,125 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:31:57,126 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:57,127 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=4, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:57,127 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:57,127 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:31:57,127 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:31:57,129 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:31:57,130 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:31:57,131 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,132 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,133 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,134 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b56, negotiated timeout = 5000
2015-12-16 17:31:57,134 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:57,135 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,135 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,136 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,137 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:57,137 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:57,151 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,152 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,153 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,154 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,155 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,156 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,157 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:57,159 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:57,160 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,172 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,174 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,175 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:57,187 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b56, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265405518,0 request:: null response:: null
2015-12-16 17:31:57,187 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b56
2015-12-16 17:31:57,188 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b56 : Unable to read additional data from server sessionid 0x251705aced21b56, likely server has closed socket
2015-12-16 17:31:57,188 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b56 closed
2015-12-16 17:31:57,188 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 63379757ns
2015-12-16 17:31:57,189 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:57,189 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:57,190 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:57,190 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:58,149 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:58,150 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:58,151 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:31:58,152 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:58,153 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:31:58,155 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:58,156 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:31:58,156 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:31:58,158 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b57, negotiated timeout = 5000
2015-12-16 17:31:58,159 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:58,159 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:58,159 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:58,159 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:31:58,160 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:58,164 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,165 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,166 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,167 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,167 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:58,169 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:58,170 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,171 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,172 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:58,172 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:58,172 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,173 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b57
2015-12-16 17:31:58,173 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b57
2015-12-16 17:31:58,173 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,174 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,175 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:31:58,177 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b57, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265405800,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:31:58,179 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b57, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265405800,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:31:58,180 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b57, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265405801,0 request:: null response:: null
2015-12-16 17:31:58,180 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b57 : Unable to read additional data from server sessionid 0x251705aced21b57, likely server has closed socket
2015-12-16 17:31:58,180 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b57
2015-12-16 17:31:58,181 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b57 closed
2015-12-16 17:31:58,181 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 8351655ns
2015-12-16 17:31:58,181 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:58,181 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:58,183 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:31:58,184 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:58,184 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #5, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 5, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:31:58,184 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:31:58,185 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@1357384 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384)), so the following Deferred is getting paused: Deferred@30100661(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:31:58,185 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:31:58,185 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:58,185 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:31:58,185 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:31:58,186 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:31:58,186 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:58,186 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:58,187 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:31:58,187 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=5, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=5, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:58,188 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:31:58,188 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:31:58,189 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,190 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,191 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,191 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b58, negotiated timeout = 5000
2015-12-16 17:31:58,192 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:58,192 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,192 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,193 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,193 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:58,194 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:58,198 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,198 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,199 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,200 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,201 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,201 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,202 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:58,204 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:58,205 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,206 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,206 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,207 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:58,210 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b58, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265405809,0 request:: null response:: null
2015-12-16 17:31:58,210 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b58
2015-12-16 17:31:58,211 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b58 : Unable to read additional data from server sessionid 0x251705aced21b58, likely server has closed socket
2015-12-16 17:31:58,211 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b58 closed
2015-12-16 17:31:58,211 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 25729511ns
2015-12-16 17:31:58,211 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:58,211 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:58,212 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:58,212 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:59,208 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:59,209 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:59,211 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:59,212 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:31:59,213 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:59,214 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:31:59,215 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:31:59,217 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b59, negotiated timeout = 5000
2015-12-16 17:31:59,217 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:59,217 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:59,223 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:59,225 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:59,226 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:31:59,226 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:59,226 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:31:59,227 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:59,228 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:59,228 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:59,228 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b59
2015-12-16 17:31:59,228 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b59
2015-12-16 17:31:59,230 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b59, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265406094,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:31:59,231 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b59, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265406094,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:31:59,232 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b59, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265406095,0 request:: null response:: null
2015-12-16 17:31:59,233 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b59
2015-12-16 17:31:59,233 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b59 closed
2015-12-16 17:31:59,233 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 5177575ns
2015-12-16 17:31:59,233 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:59,234 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:31:59,237 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:31:59,238 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:31:59,250 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:31:59,250 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:31:59,261 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #6, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 6, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:31:59,261 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:31:59,262 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:31:59,262 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@6446697 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697)), so the following Deferred is getting paused: Deferred@2388206(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:31:59,262 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:31:59,262 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:31:59,263 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:31:59,263 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:31:59,263 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:31:59,263 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:31:59,265 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=6, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=6, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:31:59,265 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:31:59,266 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bda5, negotiated timeout = 5000
2015-12-16 17:31:59,266 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:59,266 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,267 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,267 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:31:59,268 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:31:59,274 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,275 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:31:59,277 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:31:59,277 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,278 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,279 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,279 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,280 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,281 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,282 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:31:59,284 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda5, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265406125,0 request:: null response:: null
2015-12-16 17:31:59,284 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bda5
2015-12-16 17:31:59,285 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bda5 : Unable to read additional data from server sessionid 0x151705ac5a7bda5, likely server has closed socket
2015-12-16 17:31:59,285 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bda5 closed
2015-12-16 17:31:59,285 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 22189911ns
2015-12-16 17:31:59,285 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:31:59,286 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:31:59,286 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:31:59,286 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:00,308 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=5, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:00,309 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:00,324 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:00,325 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:00,327 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:00,328 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:00,329 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:00,329 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:00,339 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec239, negotiated timeout = 5000
2015-12-16 17:32:00,340 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:00,341 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:00,341 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:00,341 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:00,343 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:00,348 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,349 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,351 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,352 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,353 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:00,355 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:00,357 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,372 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,373 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:00,374 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:00,374 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,374 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec239
2015-12-16 17:32:00,375 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec239
2015-12-16 17:32:00,375 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,376 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,378 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:00,391 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec239, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265406535,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:00,394 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec239, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265406535,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:00,395 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec239, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265406536,0 request:: null response:: null
2015-12-16 17:32:00,395 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec239
2015-12-16 17:32:00,396 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec239 : Unable to read additional data from server sessionid 0x351705ac89ec239, likely server has closed socket
2015-12-16 17:32:00,396 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec239 closed
2015-12-16 17:32:00,397 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 22419446ns
2015-12-16 17:32:00,397 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:00,397 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:00,410 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:00,411 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:00,412 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:00,420 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:00,421 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:00,421 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:00,422 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #7, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 7, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:00,423 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@28625395 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395)), so the following Deferred is getting paused: Deferred@3577952(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:32:00,423 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:00,424 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b5a, negotiated timeout = 5000
2015-12-16 17:32:00,424 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:00,424 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b5a
2015-12-16 17:32:00,424 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:00,425 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:00,425 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:00,425 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:00,425 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b5a
2015-12-16 17:32:00,426 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:00,426 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=7, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=7, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:00,427 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:00,430 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:00,432 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:00,433 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:00,435 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:00,436 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:00,437 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:00,438 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:00,450 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:00,451 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:00,454 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5a, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265406557,0 request:: null response:: null
2015-12-16 17:32:00,454 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b5a
2015-12-16 17:32:00,454 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b5a : Unable to read additional data from server sessionid 0x251705aced21b5a, likely server has closed socket
2015-12-16 17:32:00,454 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b5a closed
2015-12-16 17:32:00,455 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 30305549ns
2015-12-16 17:32:00,455 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:00,455 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:00,456 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:00,456 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:01,509 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=6, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:01,509 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:01,510 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:01,511 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:01,511 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:01,513 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:01,514 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:01,514 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:01,517 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b5b, negotiated timeout = 5000
2015-12-16 17:32:01,518 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:01,518 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:01,523 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:01,524 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:01,524 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:01,525 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:01,526 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:01,527 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:01,527 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:01,527 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:01,527 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b5b
2015-12-16 17:32:01,527 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b5b
2015-12-16 17:32:01,527 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:01,529 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5b, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265407225,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:01,530 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5b, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265407225,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:01,531 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5b, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265407226,0 request:: null response:: null
2015-12-16 17:32:01,531 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b5b : Unable to read additional data from server sessionid 0x251705aced21b5b, likely server has closed socket
2015-12-16 17:32:01,532 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b5b
2015-12-16 17:32:01,532 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b5b closed
2015-12-16 17:32:01,532 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 5010761ns
2015-12-16 17:32:01,532 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:01,533 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:01,557 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:01,557 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:01,558 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:01,559 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:01,559 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:01,560 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:01,560 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #8, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 8, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:01,561 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:01,561 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@5961055 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055)), so the following Deferred is getting paused: Deferred@27480577(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:32:01,561 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:01,562 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:01,562 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:01,562 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:01,562 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:01,563 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=8, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=8, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:01,564 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:01,567 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec23a, negotiated timeout = 5000
2015-12-16 17:32:01,568 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:01,568 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,569 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,569 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:01,569 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:01,580 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,593 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:01,595 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:01,596 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,596 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,597 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,597 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,598 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:01,599 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23a, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265407269,0 request:: null response:: null
2015-12-16 17:32:01,600 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec23a
2015-12-16 17:32:01,600 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec23a : Unable to read additional data from server sessionid 0x351705ac89ec23a, likely server has closed socket
2015-12-16 17:32:01,600 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec23a closed
2015-12-16 17:32:01,600 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 38769521ns
2015-12-16 17:32:01,601 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:01,601 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:01,601 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:01,601 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:02,708 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=7, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:02,709 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:02,711 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:02,713 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:02,714 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:02,715 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:02,716 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:02,716 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:02,719 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bda6, negotiated timeout = 5000
2015-12-16 17:32:02,720 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:02,720 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:02,720 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:02,720 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:02,721 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:02,724 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,725 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,726 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,726 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,727 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:02,729 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:02,729 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,730 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,730 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:02,731 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:02,731 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,731 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bda6
2015-12-16 17:32:02,731 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bda6
2015-12-16 17:32:02,732 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,732 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,733 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,734 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,735 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,735 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,736 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,737 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:02,738 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,739 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda6, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265407562,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:02,741 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda6, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265407562,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:02,741 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda6, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265407563,0 request:: null response:: null
2015-12-16 17:32:02,742 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bda6
2015-12-16 17:32:02,742 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bda6 : Unable to read additional data from server sessionid 0x151705ac5a7bda6, likely server has closed socket
2015-12-16 17:32:02,742 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bda6 closed
2015-12-16 17:32:02,742 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 11211849ns
2015-12-16 17:32:02,743 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:02,743 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:02,744 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:02,745 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #9, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 9, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:02,745 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:02,746 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:02,746 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@6172445 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055) -> (continuation of Deferred@23535164 after retry RPC@6172445), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055) -> (continuation of Deferred@23535164 after retry RPC@6172445)), so the following Deferred is getting paused: Deferred@23535164(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:32:02,746 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:02,746 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:02,747 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:02,747 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:02,747 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:02,748 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:02,747 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:02,749 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=9, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=9, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:02,749 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:02,750 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:02,750 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:02,752 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,754 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,755 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,756 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bda7, negotiated timeout = 5000
2015-12-16 17:32:02,757 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:02,757 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,759 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,760 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,761 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:02,762 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:02,768 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,770 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,771 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,773 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,774 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:02,778 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:02,790 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,790 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,791 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,792 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,792 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,793 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:02,803 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda7, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265407576,0 request:: null response:: null
2015-12-16 17:32:02,803 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bda7 : Unable to read additional data from server sessionid 0x151705ac5a7bda7, likely server has closed socket
2015-12-16 17:32:02,806 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bda7
2015-12-16 17:32:02,806 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bda7 closed
2015-12-16 17:32:02,807 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 59712781ns
2015-12-16 17:32:02,807 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:02,807 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:02,807 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:02,808 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:04,009 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=8, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:04,010 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:04,011 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:04,012 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:04,013 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:04,014 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:04,015 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:04,015 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:04,017 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec23b, negotiated timeout = 5000
2015-12-16 17:32:04,018 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:04,018 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:04,018 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:04,019 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:04,020 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:04,023 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,024 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,025 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,026 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,027 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:04,029 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:04,030 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,031 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,031 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:04,031 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:04,032 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,032 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec23b
2015-12-16 17:32:04,032 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec23b
2015-12-16 17:32:04,033 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,034 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,034 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:04,036 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23b, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265407816,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:04,038 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23b, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265407816,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:04,039 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23b, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265407817,0 request:: null response:: null
2015-12-16 17:32:04,039 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec23b
2015-12-16 17:32:04,039 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec23b : Unable to read additional data from server sessionid 0x351705ac89ec23b, likely server has closed socket
2015-12-16 17:32:04,040 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec23b closed
2015-12-16 17:32:04,040 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 8475495ns
2015-12-16 17:32:04,040 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:04,041 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:04,042 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:04,043 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:04,043 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #10, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 10, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:04,044 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:04,044 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@29749755 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055) -> (continuation of Deferred@23535164 after retry RPC@6172445) -> (continuation of Deferred@9556517 after retry RPC@29749755), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055) -> (continuation of Deferred@23535164 after retry RPC@6172445) -> (continuation of Deferred@9556517 after retry RPC@29749755)), so the following Deferred is getting paused: Deferred@9556517(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:32:04,044 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:04,045 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:04,045 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:04,045 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:04,045 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:04,045 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:04,047 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:04,047 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:04,048 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:04,048 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=10, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=10, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:04,049 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:04,049 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,050 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,051 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,051 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b5c, negotiated timeout = 5000
2015-12-16 17:32:04,052 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:04,052 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,053 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,053 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,054 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:04,054 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:04,057 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,058 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,059 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,060 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,060 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,061 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,062 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:04,064 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:04,064 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,065 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,066 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,067 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:04,069 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5c, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265407823,0 request:: null response:: null
2015-12-16 17:32:04,070 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b5c
2015-12-16 17:32:04,070 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b5c : Unable to read additional data from server sessionid 0x251705aced21b5c, likely server has closed socket
2015-12-16 17:32:04,070 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b5c closed
2015-12-16 17:32:04,071 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 25418840ns
2015-12-16 17:32:04,071 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:04,071 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:04,072 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:04,072 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:05,569 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=9, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:05,569 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:05,572 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:05,572 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:05,573 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:05,574 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:05,574 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:05,575 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:05,577 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec23c, negotiated timeout = 5000
2015-12-16 17:32:05,577 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:05,577 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:05,577 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:05,577 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:05,580 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:05,585 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,586 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,586 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,587 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,587 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:05,589 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:05,589 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,590 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,590 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:05,590 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:05,591 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec23c
2015-12-16 17:32:05,591 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,591 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec23c
2015-12-16 17:32:05,591 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,592 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,592 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:05,594 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23c, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265408159,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:05,595 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23c, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265408159,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:05,595 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23c, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265408160,0 request:: null response:: null
2015-12-16 17:32:05,596 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec23c
2015-12-16 17:32:05,596 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec23c : Unable to read additional data from server sessionid 0x351705ac89ec23c, likely server has closed socket
2015-12-16 17:32:05,596 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec23c closed
2015-12-16 17:32:05,596 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 5657968ns
2015-12-16 17:32:05,596 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:05,597 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:05,602 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:05,605 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #11, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 11, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:05,605 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:05,605 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:05,605 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@210991 returned Deferred@1981014(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055) -> (continuation of Deferred@23535164 after retry RPC@6172445) -> (continuation of Deferred@9556517 after retry RPC@29749755) -> (continuation of Deferred@22912742 after retry RPC@210991), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@8916011 after retry RPC@21597776) -> (continuation of Deferred@4531348 after retry RPC@24798246) -> (continuation of Deferred@19136330 after retry RPC@31521182) -> (continuation of Deferred@30100661 after retry RPC@1357384) -> (continuation of Deferred@2388206 after retry RPC@6446697) -> (continuation of Deferred@3577952 after retry RPC@28625395) -> (continuation of Deferred@27480577 after retry RPC@5961055) -> (continuation of Deferred@23535164 after retry RPC@6172445) -> (continuation of Deferred@9556517 after retry RPC@29749755) -> (continuation of Deferred@22912742 after retry RPC@210991)), so the following Deferred is getting paused: Deferred@22912742(state=PAUSED, result=Deferred@1981014, callback=<none>, errback=<none>)
2015-12-16 17:32:05,605 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:05,606 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:05,606 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:05,606 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:05,606 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:05,606 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:05,607 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:05,608 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=11, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:05,608 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:05,619 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:05,620 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:05,621 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,621 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,621 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,622 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,622 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,623 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,623 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bda8, negotiated timeout = 5000
2015-12-16 17:32:05,623 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:05,624 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,624 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,625 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,625 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:05,625 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:05,627 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,628 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,628 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,629 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,629 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,630 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,630 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:05,632 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:05,645 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,645 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,646 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,646 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,647 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,647 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:05,649 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda8, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265408172,0 request:: null response:: null
2015-12-16 17:32:05,650 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bda8
2015-12-16 17:32:05,650 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bda8 : Unable to read additional data from server sessionid 0x151705ac5a7bda8, likely server has closed socket
2015-12-16 17:32:05,650 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bda8 closed
2015-12-16 17:32:05,650 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 44435770ns
2015-12-16 17:32:05,651 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:05,651 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:05,652 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:05,652 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:07,649 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=10, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:07,650 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:07,662 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:07,663 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:07,663 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:07,664 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:07,665 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:07,666 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:07,674 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bda9, negotiated timeout = 5000
2015-12-16 17:32:07,675 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:07,675 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:07,685 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:07,686 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:07,687 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:07,688 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:07,690 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,689 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:07,691 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,691 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:07,692 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:07,692 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bda9
2015-12-16 17:32:07,692 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bda9
2015-12-16 17:32:07,693 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,694 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,694 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:07,695 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,695 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,698 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:07,700 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda9, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265408544,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:07,701 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda9, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265408544,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:07,702 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bda9, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265408545,0 request:: null response:: null
2015-12-16 17:32:07,702 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bda9
2015-12-16 17:32:07,702 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bda9 : Unable to read additional data from server sessionid 0x151705ac5a7bda9, likely server has closed socket
2015-12-16 17:32:07,702 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bda9 closed
2015-12-16 17:32:07,702 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 10316401ns
2015-12-16 17:32:07,702 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:07,703 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:07,704 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:07,705 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:07,705 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:07,706 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:07,707 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:07,707 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:07,710 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b5d, negotiated timeout = 5000
2015-12-16 17:32:07,710 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:07,710 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:07,711 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:07,711 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:07,712 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:07,714 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,714 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,715 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Probe Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) failed
org.hbase.async.NonRecoverableException: Too many attempts: Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3632) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) [zookeeper_client-3.4.5.jar:3.4.5-1461890]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [zookeeper_client-3.4.5.jar:3.4.5-1461890]
2015-12-16 17:32:07,716 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Retrying 3 RPCs on NSREd region "hbase:meta,,1"
2015-12-16 17:32:07,716 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,717 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #12, payload=BigEndianHeapChannelBuffer(ridx=12, widx=159, cap=159) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x8F\t\x08\x0C\x1A\x03Get \x01\x83\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12n\nbperf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:32:07,717 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,717 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #13, payload=BigEndianHeapChannelBuffer(ridx=12, widx=163, cap=163) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x93\t\x08\x0D\x1A\x03Get \x01\x87\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12r\nfperf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:32:07,717 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 147
2015-12-16 17:32:07,718 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:07,718 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:07,718 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 151
2015-12-16 17:32:07,719 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:07,719 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:07,720 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=12, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=1, timeout=0, hasTimedout=false)
2015-12-16 17:32:07,720 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) seems to be splitting or closing it.
2015-12-16 17:32:07,720 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:07,721 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:07,721 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:07,721 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=13, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=1, timeout=0, hasTimedout=false)
2015-12-16 17:32:07,724 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b5d
2015-12-16 17:32:07,725 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,726 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b5d
2015-12-16 17:32:07,726 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,727 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:07,729 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,730 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,731 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:07,732 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,733 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:07,734 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:07,736 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5d, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265408557,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:07,738 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5d, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265408557,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:07,739 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5d, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265408558,0 request:: null response:: null
2015-12-16 17:32:07,739 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b5d
2015-12-16 17:32:07,739 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b5d : Unable to read additional data from server sessionid 0x251705aced21b5d, likely server has closed socket
2015-12-16 17:32:07,739 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b5d closed
2015-12-16 17:32:07,740 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 15371514ns
2015-12-16 17:32:07,740 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:07,740 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:07,740 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:08,129 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=null)
2015-12-16 17:32:08,129 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:08,130 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:08,131 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:08,131 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:08,132 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:08,133 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:08,133 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:08,136 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdaa, negotiated timeout = 5000
2015-12-16 17:32:08,136 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:08,136 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,136 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:08,136 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:08,137 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,139 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,140 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,141 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,142 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,142 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:08,143 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:08,144 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,145 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,145 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:08,145 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:08,146 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdaa
2015-12-16 17:32:08,146 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdaa
2015-12-16 17:32:08,146 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,147 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,147 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,148 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,149 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdaa, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265408643,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:08,150 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdaa, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265408643,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:08,151 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdaa, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265408644,0 request:: null response:: null
2015-12-16 17:32:08,151 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdaa
2015-12-16 17:32:08,151 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdaa : Unable to read additional data from server sessionid 0x151705ac5a7bdaa, likely server has closed socket
2015-12-16 17:32:08,151 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdaa closed
2015-12-16 17:32:08,151 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 5785898ns
2015-12-16 17:32:08,151 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:08,152 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:08,152 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:08,153 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #14, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 14, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:08,153 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:08,153 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:08,154 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@3286473 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473)), so the following Deferred is getting paused: Deferred@10154658(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:08,154 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:08,154 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:08,154 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:08,154 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:08,154 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:08,155 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:08,155 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:08,155 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:08,155 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:08,156 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=14, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:08,156 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:08,156 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,157 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,157 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,157 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,158 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,158 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdab, negotiated timeout = 5000
2015-12-16 17:32:08,158 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,159 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,159 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,159 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,159 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:08,160 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:08,162 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,162 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,162 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,163 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,163 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,163 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,164 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,164 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,164 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:08,165 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:08,166 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,166 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,167 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,167 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,167 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,168 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,168 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,168 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,170 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdab, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265408648,0 request:: null response:: null
2015-12-16 17:32:08,170 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdab
2015-12-16 17:32:08,170 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdab : Unable to read additional data from server sessionid 0x151705ac5a7bdab, likely server has closed socket
2015-12-16 17:32:08,171 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdab closed
2015-12-16 17:32:08,171 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 16604166ns
2015-12-16 17:32:08,171 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:08,171 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,171 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:08,171 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:08,769 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=1, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:08,769 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:08,783 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:08,784 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:08,785 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:08,787 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:08,793 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:08,794 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:08,796 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec23d, negotiated timeout = 5000
2015-12-16 17:32:08,796 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:08,796 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:08,797 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,797 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:08,804 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,806 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,807 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,808 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,809 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,810 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:08,812 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:08,813 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,814 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,814 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:08,814 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:08,815 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec23d
2015-12-16 17:32:08,815 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,815 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec23d
2015-12-16 17:32:08,816 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,817 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,818 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,830 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23d, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265408781,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:08,832 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23d, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265408781,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:08,833 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23d, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265408782,0 request:: null response:: null
2015-12-16 17:32:08,833 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec23d
2015-12-16 17:32:08,833 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec23d : Unable to read additional data from server sessionid 0x351705ac89ec23d, likely server has closed socket
2015-12-16 17:32:08,834 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec23d closed
2015-12-16 17:32:08,834 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 19377451ns
2015-12-16 17:32:08,834 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:08,834 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:08,849 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:08,850 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:08,851 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:08,852 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:08,852 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:08,855 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdac, negotiated timeout = 5000
2015-12-16 17:32:08,856 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:08,856 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:08,861 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:08,863 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:08,864 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,864 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:08,864 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:08,865 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:08,865 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:08,866 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:08,866 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdac
2015-12-16 17:32:08,866 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdac
2015-12-16 17:32:08,867 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #15, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 15, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:08,867 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@32365523 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523)), so the following Deferred is getting paused: Deferred@16462868(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:08,867 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:08,869 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:08,869 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:08,870 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=15, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:08,870 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:08,873 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,874 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:08,875 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:08,877 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdac, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265408797,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:08,878 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdac, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265408797,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:08,879 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdac, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265408800,0 request:: null response:: null
2015-12-16 17:32:08,880 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdac
2015-12-16 17:32:08,880 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdac : Unable to read additional data from server sessionid 0x151705ac5a7bdac, likely server has closed socket
2015-12-16 17:32:08,880 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdac closed
2015-12-16 17:32:08,880 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 14454853ns
2015-12-16 17:32:08,881 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:08,881 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:08,881 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:09,688 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:09,689 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:09,690 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:09,691 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:09,692 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:09,702 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:09,703 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:09,703 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:09,705 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec23e, negotiated timeout = 5000
2015-12-16 17:32:09,706 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:09,706 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:09,711 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:09,712 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:09,712 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:09,718 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,719 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,719 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:09,721 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:09,722 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,722 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,723 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:09,723 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:09,723 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,723 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec23e
2015-12-16 17:32:09,724 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec23e
2015-12-16 17:32:09,724 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,724 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,725 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:09,726 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,727 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23e, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265409010,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:09,728 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23e, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265409010,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:09,729 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23e, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265409011,0 request:: null response:: null
2015-12-16 17:32:09,729 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec23e
2015-12-16 17:32:09,730 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec23e closed
2015-12-16 17:32:09,730 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 6531558ns
2015-12-16 17:32:09,730 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:09,730 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:09,732 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:09,739 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:09,739 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #16, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 16, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:09,740 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:09,740 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@20432763 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763)), so the following Deferred is getting paused: Deferred@17395802(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:09,740 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:09,740 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:09,741 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:09,741 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:09,741 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:09,741 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:09,741 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:09,742 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=16, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:09,743 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:09,743 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:09,744 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:09,745 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,745 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,746 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,746 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec23f, negotiated timeout = 5000
2015-12-16 17:32:09,747 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:09,748 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,748 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,749 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,749 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:09,750 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:09,753 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,755 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,755 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,756 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,756 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:09,758 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:09,759 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,759 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,760 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,760 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:09,762 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec23f, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265409020,0 request:: null response:: null
2015-12-16 17:32:09,762 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec23f
2015-12-16 17:32:09,762 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec23f : Unable to read additional data from server sessionid 0x351705ac89ec23f, likely server has closed socket
2015-12-16 17:32:09,763 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec23f closed
2015-12-16 17:32:09,763 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 22586194ns
2015-12-16 17:32:09,763 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:09,763 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:09,764 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:09,764 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:10,749 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:10,749 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:10,751 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:10,752 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:10,752 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:10,753 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:10,754 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:10,755 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:10,757 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b5e, negotiated timeout = 5000
2015-12-16 17:32:10,758 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:10,758 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:10,758 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:10,758 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:10,759 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:10,762 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,763 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,764 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,765 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,765 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:10,767 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:10,767 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,768 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,769 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:10,769 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:10,769 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,769 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b5e
2015-12-16 17:32:10,770 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b5e
2015-12-16 17:32:10,770 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,771 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,771 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:10,773 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5e, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265409298,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:10,775 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5e, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265409298,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:10,775 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5e, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265409299,0 request:: null response:: null
2015-12-16 17:32:10,776 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b5e
2015-12-16 17:32:10,776 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b5e : Unable to read additional data from server sessionid 0x251705aced21b5e, likely server has closed socket
2015-12-16 17:32:10,776 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b5e closed
2015-12-16 17:32:10,776 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 7091950ns
2015-12-16 17:32:10,777 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:10,777 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:10,778 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:10,779 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #17, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 17, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:10,779 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:10,779 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@4534563 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563)), so the following Deferred is getting paused: Deferred@10286092(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:10,779 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:10,779 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:10,779 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:10,780 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:10,780 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:10,780 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:10,780 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:10,781 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:10,781 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=17, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=5, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:10,781 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:10,782 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:10,782 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:10,783 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,783 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,784 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,784 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,785 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec240, negotiated timeout = 5000
2015-12-16 17:32:10,785 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:10,785 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,786 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,786 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,787 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:10,787 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:10,790 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,790 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,791 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,791 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,792 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,792 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,793 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:10,794 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:10,795 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,795 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,796 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,796 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,797 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,798 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:10,799 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec240, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265409306,0 request:: null response:: null
2015-12-16 17:32:10,800 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec240
2015-12-16 17:32:10,800 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec240 : Unable to read additional data from server sessionid 0x351705ac89ec240, likely server has closed socket
2015-12-16 17:32:10,800 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec240 closed
2015-12-16 17:32:10,801 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 20724300ns
2015-12-16 17:32:10,801 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:10,801 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:10,801 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:10,802 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:11,809 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:11,809 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:11,812 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:11,813 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:11,815 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:11,820 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:11,829 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:11,830 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:11,840 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec241, negotiated timeout = 5000
2015-12-16 17:32:11,841 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:11,841 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:11,842 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:11,842 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:11,843 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:11,848 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,849 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,850 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,851 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,852 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:11,855 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:11,856 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,857 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,858 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:11,858 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:11,859 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec241
2015-12-16 17:32:11,859 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,859 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec241
2015-12-16 17:32:11,868 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,872 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,874 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:11,880 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec241, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265409485,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:11,882 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec241, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265409485,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:11,883 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec241, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265409486,0 request:: null response:: null
2015-12-16 17:32:11,883 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec241
2015-12-16 17:32:11,884 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec241 : Unable to read additional data from server sessionid 0x351705ac89ec241, likely server has closed socket
2015-12-16 17:32:11,884 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec241 closed
2015-12-16 17:32:11,895 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 36921290ns
2015-12-16 17:32:11,896 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:11,896 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:11,911 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:11,911 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:11,913 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:11,919 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:11,921 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:11,922 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #18, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 18, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:11,922 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:11,922 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@14650173 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173)), so the following Deferred is getting paused: Deferred@3051728(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:11,923 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:11,923 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:11,924 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:11,924 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:11,924 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:11,924 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:11,925 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b5f, negotiated timeout = 5000
2015-12-16 17:32:11,925 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:11,926 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:11,942 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=18, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=6, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:11,943 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:11,944 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:11,946 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:11,947 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:11,948 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:11,951 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:11,952 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:11,953 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:11,954 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:11,957 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b5f, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265409498,0 request:: null response:: null
2015-12-16 17:32:11,958 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b5f
2015-12-16 17:32:11,958 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b5f : Unable to read additional data from server sessionid 0x251705aced21b5f, likely server has closed socket
2015-12-16 17:32:11,958 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b5f closed
2015-12-16 17:32:11,958 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 34883607ns
2015-12-16 17:32:11,959 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:11,959 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:11,959 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:11,960 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:12,988 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=5, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:12,989 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:12,991 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:12,996 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:12,996 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:12,998 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:13,000 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:13,000 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:13,004 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdad, negotiated timeout = 5000
2015-12-16 17:32:13,004 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:13,005 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:13,010 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:13,010 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:13,014 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:13,024 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,025 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,026 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:13,029 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:13,031 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,034 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,037 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:13,037 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:13,037 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,037 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdad
2015-12-16 17:32:13,038 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdad
2015-12-16 17:32:13,038 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,040 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,041 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,053 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdad, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265409775,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:13,055 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdad, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265409775,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:13,057 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdad, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265409777,0 request:: null response:: null
2015-12-16 17:32:13,057 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdad
2015-12-16 17:32:13,057 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdad : Unable to read additional data from server sessionid 0x151705ac5a7bdad, likely server has closed socket
2015-12-16 17:32:13,058 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdad closed
2015-12-16 17:32:13,058 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 20718750ns
2015-12-16 17:32:13,058 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:13,059 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:13,062 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:13,062 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:13,064 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:13,065 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:13,066 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:13,068 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b60, negotiated timeout = 5000
2015-12-16 17:32:13,069 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:13,069 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:13,070 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:13,078 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:13,079 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #19, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 19, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:13,079 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:13,080 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@21850043 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043)), so the following Deferred is getting paused: Deferred@13976679(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:13,080 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:13,081 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:13,082 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:13,082 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:13,082 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:13,082 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b60
2015-12-16 17:32:13,083 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b60
2015-12-16 17:32:13,083 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=19, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=7, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:13,084 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:13,087 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,090 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,092 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:13,096 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,099 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,101 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:13,103 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:13,110 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:13,115 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,118 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,121 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:13,122 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:13,124 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,127 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,129 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:13,132 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,136 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:13,138 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:13,143 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b60, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265409808,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:13,149 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b60, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265409809,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:13,152 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b60, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265409810,0 request:: null response:: null
2015-12-16 17:32:13,153 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b60
2015-12-16 17:32:13,154 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b60 : Unable to read additional data from server sessionid 0x251705aced21b60, likely server has closed socket
2015-12-16 17:32:13,154 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b60 closed
2015-12-16 17:32:13,155 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 72807811ns
2015-12-16 17:32:13,156 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:13,156 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:13,157 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:14,169 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=6, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:14,169 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:14,170 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:14,171 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:14,172 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:14,173 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:14,174 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:14,175 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:14,177 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec242, negotiated timeout = 5000
2015-12-16 17:32:14,178 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:14,178 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:14,178 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:14,178 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:14,179 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:14,182 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,182 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,183 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,184 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,184 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:14,186 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:14,187 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,187 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,188 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:14,188 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:14,188 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec242
2015-12-16 17:32:14,188 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,189 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec242
2015-12-16 17:32:14,189 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,190 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,190 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:14,192 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec242, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265410161,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:14,193 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec242, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265410161,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:14,194 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec242, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265410162,0 request:: null response:: null
2015-12-16 17:32:14,194 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec242
2015-12-16 17:32:14,194 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec242 : Unable to read additional data from server sessionid 0x351705ac89ec242, likely server has closed socket
2015-12-16 17:32:14,195 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec242 closed
2015-12-16 17:32:14,195 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 6596395ns
2015-12-16 17:32:14,195 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:14,195 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:14,197 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:14,197 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #20, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 20, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:14,197 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:14,198 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@27991727 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727)), so the following Deferred is getting paused: Deferred@23833448(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:14,198 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:14,198 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:14,198 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:14,199 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:14,199 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:14,199 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:14,199 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:14,199 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:14,200 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=20, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=8, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:14,200 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:14,200 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:14,201 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:14,202 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,203 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,203 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,204 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,204 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,205 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdae, negotiated timeout = 5000
2015-12-16 17:32:14,205 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:14,206 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,206 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,207 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,207 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:14,207 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:14,210 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,211 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,212 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,212 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,213 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,213 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,214 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:14,215 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:14,216 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,217 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,217 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,218 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,218 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,219 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:14,221 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdae, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265410168,0 request:: null response:: null
2015-12-16 17:32:14,221 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdae
2015-12-16 17:32:14,222 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdae : Unable to read additional data from server sessionid 0x151705ac5a7bdae, likely server has closed socket
2015-12-16 17:32:14,222 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdae closed
2015-12-16 17:32:14,222 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 23365850ns
2015-12-16 17:32:14,222 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:14,222 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:14,223 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:14,223 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:15,348 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=7, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:15,349 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:15,350 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:15,351 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:15,351 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:15,352 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:15,353 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:15,353 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:15,356 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb0, negotiated timeout = 5000
2015-12-16 17:32:15,357 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:15,357 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:15,357 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:15,357 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:15,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:15,362 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,363 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,364 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,364 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,365 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:15,367 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:15,368 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,369 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,369 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:15,370 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:15,370 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,370 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdb0
2015-12-16 17:32:15,370 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdb0
2015-12-16 17:32:15,371 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,372 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,372 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:15,374 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb0, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265410450,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:15,376 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb0, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265410450,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:15,377 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb0, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265410451,0 request:: null response:: null
2015-12-16 17:32:15,378 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb0
2015-12-16 17:32:15,378 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb0 closed
2015-12-16 17:32:15,378 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 8170972ns
2015-12-16 17:32:15,378 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:15,379 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:15,381 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:15,382 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:15,383 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:15,384 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:15,385 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:15,387 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec245, negotiated timeout = 5000
2015-12-16 17:32:15,387 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:15,388 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:15,392 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:15,394 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:15,398 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:15,398 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:15,398 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:15,401 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:15,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #21, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 21, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:15,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:15,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:15,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@23963784 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727) -> (continuation of Deferred@5473730 after retry RPC@23963784), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727) -> (continuation of Deferred@5473730 after retry RPC@23963784)), so the following Deferred is getting paused: Deferred@5473730(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:15,402 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:15,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec245
2015-12-16 17:32:15,403 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec245
2015-12-16 17:32:15,405 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:15,405 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec245, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265410475,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:15,405 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:15,406 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=21, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=9, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:15,406 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:15,406 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec245, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265410475,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:15,407 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec245, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265410477,0 request:: null response:: null
2015-12-16 17:32:15,408 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec245
2015-12-16 17:32:15,408 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec245 : Unable to read additional data from server sessionid 0x351705ac89ec245, likely server has closed socket
2015-12-16 17:32:15,408 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec245 closed
2015-12-16 17:32:15,409 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 6046331ns
2015-12-16 17:32:15,409 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:15,409 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:15,409 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:16,669 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=8, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:16,669 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:16,682 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:16,682 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:16,683 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:16,684 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:16,684 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:16,687 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b63, negotiated timeout = 5000
2015-12-16 17:32:16,687 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:16,687 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:16,693 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:16,693 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:16,693 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:16,694 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:16,695 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,696 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,698 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,698 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,698 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:16,700 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:16,700 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,701 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,701 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:16,701 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:16,701 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,701 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b63
2015-12-16 17:32:16,702 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b63
2015-12-16 17:32:16,702 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,702 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,703 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,703 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,703 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,704 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,704 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,705 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,705 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:16,705 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,709 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b63, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265410843,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:16,710 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b63, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265410843,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:16,712 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b63, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265410844,0 request:: null response:: null
2015-12-16 17:32:16,712 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b63
2015-12-16 17:32:16,713 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b63 closed
2015-12-16 17:32:16,713 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 11672115ns
2015-12-16 17:32:16,713 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:16,714 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:16,729 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:16,737 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #22, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 22, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:16,737 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@16720010 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727) -> (continuation of Deferred@5473730 after retry RPC@23963784) -> (continuation of Deferred@28401723 after retry RPC@16720010), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727) -> (continuation of Deferred@5473730 after retry RPC@23963784) -> (continuation of Deferred@28401723 after retry RPC@16720010)), so the following Deferred is getting paused: Deferred@28401723(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:16,738 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:16,738 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:16,738 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:16,738 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:16,739 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:16,739 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:16,739 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:16,740 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=22, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=10, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:16,740 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:16,740 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:16,749 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:16,750 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:16,751 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:16,751 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,752 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,753 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,753 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,754 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,754 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb1, negotiated timeout = 5000
2015-12-16 17:32:16,755 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:16,755 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,755 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,756 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,756 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:16,756 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:16,759 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,760 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,761 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,761 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,762 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,762 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,763 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:16,765 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:16,765 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,766 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,767 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,767 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,768 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,768 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:16,771 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb1, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265410850,0 request:: null response:: null
2015-12-16 17:32:16,771 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb1
2015-12-16 17:32:16,771 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb1 : Unable to read additional data from server sessionid 0x151705ac5a7bdb1, likely server has closed socket
2015-12-16 17:32:16,772 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb1 closed
2015-12-16 17:32:16,772 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 33854103ns
2015-12-16 17:32:16,772 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:16,772 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:16,773 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:16,773 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:18,268 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=9, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:18,269 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:18,270 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:18,271 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:18,271 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:18,272 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:18,273 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:18,274 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:18,277 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb2, negotiated timeout = 5000
2015-12-16 17:32:18,277 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:18,277 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:18,280 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:18,281 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:18,281 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:18,282 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,282 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,283 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,284 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,284 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:18,286 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:18,287 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,288 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,288 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:18,288 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:18,288 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdb2
2015-12-16 17:32:18,289 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdb2
2015-12-16 17:32:18,289 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,290 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,291 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,291 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,292 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:18,293 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,294 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb2, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265411297,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:18,295 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb2, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265411297,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:18,296 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb2, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265411298,0 request:: null response:: null
2015-12-16 17:32:18,297 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb2
2015-12-16 17:32:18,297 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb2 closed
2015-12-16 17:32:18,297 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 8555790ns
2015-12-16 17:32:18,297 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:18,297 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:18,299 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:18,299 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #23, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 23, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:18,299 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:18,300 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:18,300 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@10000461 returned Deferred@5167431(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727) -> (continuation of Deferred@5473730 after retry RPC@23963784) -> (continuation of Deferred@28401723 after retry RPC@16720010) -> (continuation of Deferred@3611610 after retry RPC@10000461), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10154658 after retry RPC@3286473) -> (continuation of Deferred@16462868 after retry RPC@32365523) -> (continuation of Deferred@17395802 after retry RPC@20432763) -> (continuation of Deferred@10286092 after retry RPC@4534563) -> (continuation of Deferred@3051728 after retry RPC@14650173) -> (continuation of Deferred@13976679 after retry RPC@21850043) -> (continuation of Deferred@23833448 after retry RPC@27991727) -> (continuation of Deferred@5473730 after retry RPC@23963784) -> (continuation of Deferred@28401723 after retry RPC@16720010) -> (continuation of Deferred@3611610 after retry RPC@10000461)), so the following Deferred is getting paused: Deferred@3611610(state=PAUSED, result=Deferred@5167431, callback=<none>, errback=<none>)
2015-12-16 17:32:18,300 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:18,310 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:18,312 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:18,313 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:18,314 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:18,314 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:18,314 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:18,315 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:18,315 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=23, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:18,315 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:18,316 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:18,316 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,316 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,317 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,318 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,319 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,319 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec246, negotiated timeout = 5000
2015-12-16 17:32:18,320 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:18,320 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,320 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,321 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,321 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:18,321 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:18,323 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,323 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,324 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,324 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,324 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,325 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,325 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,325 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,326 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:18,327 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:18,327 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,328 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,328 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,328 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,329 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,329 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:18,357 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec246, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265411329,0 request:: null response:: null
2015-12-16 17:32:18,358 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec246
2015-12-16 17:32:18,358 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec246 : Unable to read additional data from server sessionid 0x351705ac89ec246, likely server has closed socket
2015-12-16 17:32:18,358 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec246 closed
2015-12-16 17:32:18,358 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 45713939ns
2015-12-16 17:32:18,358 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:18,358 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:18,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:18,359 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:20,348 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=10, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:20,349 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:20,352 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:20,355 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:20,356 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:20,357 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:20,368 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:20,369 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:20,371 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb3, negotiated timeout = 5000
2015-12-16 17:32:20,372 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:20,372 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,372 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:20,372 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:20,373 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,375 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,376 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,377 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,378 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,378 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:20,380 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:20,381 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,381 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,382 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:20,382 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:20,382 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,382 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdb3
2015-12-16 17:32:20,382 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdb3
2015-12-16 17:32:20,382 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,383 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,383 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,384 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,384 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,385 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,385 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,386 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,386 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,387 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,388 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb3, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265412101,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:20,390 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb3, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265412101,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:20,391 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb3, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265412102,0 request:: null response:: null
2015-12-16 17:32:20,392 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb3
2015-12-16 17:32:20,392 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb3 : Unable to read additional data from server sessionid 0x151705ac5a7bdb3, likely server has closed socket
2015-12-16 17:32:20,392 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb3 closed
2015-12-16 17:32:20,394 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 11732712ns
2015-12-16 17:32:20,394 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:20,394 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:20,397 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:20,397 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:20,398 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:20,399 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:20,399 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:20,401 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb4, negotiated timeout = 5000
2015-12-16 17:32:20,402 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:20,402 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:20,404 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:20,413 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,414 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:20,414 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:20,414 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,415 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Probe Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) failed
org.hbase.async.NonRecoverableException: Too many attempts: Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3632) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) [zookeeper_client-3.4.5.jar:3.4.5-1461890]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [zookeeper_client-3.4.5.jar:3.4.5-1461890]
2015-12-16 17:32:20,416 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Retrying 3 RPCs on NSREd region "hbase:meta,,1"
2015-12-16 17:32:20,416 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:20,416 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #24, payload=BigEndianHeapChannelBuffer(ridx=12, widx=159, cap=159) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x8F\t\x08\x18\x1A\x03Get \x01\x83\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12n\nbperf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:32:20,417 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,417 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #25, payload=BigEndianHeapChannelBuffer(ridx=12, widx=163, cap=163) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x93\t\x08\x19\x1A\x03Get \x01\x87\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12r\nfperf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:32:20,417 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 147
2015-12-16 17:32:20,417 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:20,417 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,417 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 151
2015-12-16 17:32:20,418 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdb4
2015-12-16 17:32:20,418 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:20,418 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,418 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:20,418 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:20,418 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdb4
2015-12-16 17:32:20,419 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=24, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=2, timeout=0, hasTimedout=false)
2015-12-16 17:32:20,419 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,419 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) seems to be splitting or closing it.
2015-12-16 17:32:20,420 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,420 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:20,420 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:20,420 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,421 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=25, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=2, timeout=0, hasTimedout=false)
2015-12-16 17:32:20,421 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,421 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,422 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,422 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,423 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,423 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,424 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb4, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265412110,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:20,425 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb4, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265412110,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:20,426 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb4, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265412111,0 request:: null response:: null
2015-12-16 17:32:20,427 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb4
2015-12-16 17:32:20,427 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb4 : Unable to read additional data from server sessionid 0x151705ac5a7bdb4, likely server has closed socket
2015-12-16 17:32:20,427 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb4 closed
2015-12-16 17:32:20,427 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 9729111ns
2015-12-16 17:32:20,428 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:20,428 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:20,428 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:20,829 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=null)
2015-12-16 17:32:20,829 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:20,831 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:20,831 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:20,831 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:20,832 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:20,833 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:20,834 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:20,836 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec247, negotiated timeout = 5000
2015-12-16 17:32:20,836 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:20,837 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:20,839 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,840 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:20,840 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,841 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,841 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,842 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,842 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,843 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:20,845 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:20,845 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,846 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,846 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,847 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:20,847 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:20,847 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec247
2015-12-16 17:32:20,847 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec247
2015-12-16 17:32:20,848 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,849 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,849 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:20,851 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec247, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265412212,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:20,852 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec247, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265412212,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:20,852 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec247, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265412213,0 request:: null response:: null
2015-12-16 17:32:20,853 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec247
2015-12-16 17:32:20,853 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec247 closed
2015-12-16 17:32:20,853 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 6045684ns
2015-12-16 17:32:20,853 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:20,854 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:20,873 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:20,873 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:20,874 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #26, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 26, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:20,874 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:20,875 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:20,875 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@32251291 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291)), so the following Deferred is getting paused: Deferred@11831907(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:20,875 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:20,875 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:20,876 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:20,876 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:20,876 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:20,876 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:20,876 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:20,877 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,877 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:20,877 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,877 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=26, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:20,878 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,878 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:20,878 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,878 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,879 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,879 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb5, negotiated timeout = 5000
2015-12-16 17:32:20,879 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,879 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,880 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,880 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,880 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:20,881 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:20,883 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,884 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,884 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,885 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,885 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,886 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,886 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,886 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,887 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:20,889 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:20,889 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,890 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,890 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,890 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,891 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,891 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,892 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,892 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:20,900 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb5, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265412220,0 request:: null response:: null
2015-12-16 17:32:20,900 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb5
2015-12-16 17:32:20,900 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb5 : Unable to read additional data from server sessionid 0x151705ac5a7bdb5, likely server has closed socket
2015-12-16 17:32:20,901 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb5 closed
2015-12-16 17:32:20,901 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 25523764ns
2015-12-16 17:32:20,901 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:20,901 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:20,901 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:20,902 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:21,489 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=1, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:21,489 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:21,490 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:21,491 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:21,491 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:21,492 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:21,493 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:21,493 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:21,496 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb6, negotiated timeout = 5000
2015-12-16 17:32:21,496 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:21,496 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:21,496 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:21,496 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:21,497 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:21,499 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,499 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,501 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,501 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,501 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:21,503 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:21,503 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,504 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,504 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,505 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,505 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,506 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,506 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,506 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,507 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,507 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:21,509 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:21,509 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:21,509 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdb6
2015-12-16 17:32:21,509 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdb6
2015-12-16 17:32:21,509 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb6, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265412360,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:21,510 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb6, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265412360,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:21,511 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb6, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265412363,0 request:: null response:: null
2015-12-16 17:32:21,512 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb6
2015-12-16 17:32:21,512 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb6 : Unable to read additional data from server sessionid 0x151705ac5a7bdb6, likely server has closed socket
2015-12-16 17:32:21,512 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb6 closed
2015-12-16 17:32:21,512 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 3121709ns
2015-12-16 17:32:21,512 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:21,512 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:21,518 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:21,519 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #27, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 27, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:21,519 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:21,520 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@30875582 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582)), so the following Deferred is getting paused: Deferred@6620237(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:21,520 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:21,520 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:21,520 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:21,520 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:21,521 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:21,521 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:21,521 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:21,521 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=27, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:21,522 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:21,522 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:21,523 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:21,523 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:21,524 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,524 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,525 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,525 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,525 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,526 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,526 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,526 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb7, negotiated timeout = 5000
2015-12-16 17:32:21,526 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:21,527 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,527 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,527 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,527 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:21,528 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:21,530 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,530 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,531 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,531 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,531 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,532 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,532 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,533 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,533 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:21,534 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:21,535 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,535 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,535 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,536 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,536 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,536 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,537 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,537 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:21,539 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb7, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265412371,0 request:: null response:: null
2015-12-16 17:32:21,539 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb7
2015-12-16 17:32:21,539 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb7 : Unable to read additional data from server sessionid 0x151705ac5a7bdb7, likely server has closed socket
2015-12-16 17:32:21,540 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb7 closed
2015-12-16 17:32:21,540 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 19373337ns
2015-12-16 17:32:21,540 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:21,540 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:21,540 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:21,540 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:22,329 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:22,330 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:22,345 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:22,346 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:22,347 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:22,348 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:22,348 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:22,350 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec248, negotiated timeout = 5000
2015-12-16 17:32:22,350 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:22,350 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:22,354 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:22,355 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:22,358 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:22,358 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:22,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:22,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:22,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:22,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:22,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec248
2015-12-16 17:32:22,359 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec248
2015-12-16 17:32:22,360 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec248, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265412579,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:22,361 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec248, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265412579,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:22,361 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec248, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265412580,0 request:: null response:: null
2015-12-16 17:32:22,362 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec248
2015-12-16 17:32:22,362 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec248 : Unable to read additional data from server sessionid 0x351705ac89ec248, likely server has closed socket
2015-12-16 17:32:22,362 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec248 closed
2015-12-16 17:32:22,362 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 2645841ns
2015-12-16 17:32:22,362 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:22,362 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:22,376 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:22,376 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:22,383 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:22,384 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #28, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 28, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:22,384 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:22,384 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@26009708 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708)), so the following Deferred is getting paused: Deferred@31587399(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:22,385 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:22,385 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:22,385 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:22,385 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:22,385 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:22,385 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:22,386 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:22,386 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,386 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:22,386 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,387 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=28, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:22,387 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,387 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:22,387 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,388 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,388 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,388 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,389 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb8, negotiated timeout = 5000
2015-12-16 17:32:22,389 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:22,389 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,389 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,390 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,390 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:22,390 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:22,393 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,393 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,393 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,394 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,394 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,394 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,395 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,395 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,395 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:22,397 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:22,397 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,397 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,398 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,398 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,398 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,399 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,399 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,399 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:22,401 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb8, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265412588,0 request:: null response:: null
2015-12-16 17:32:22,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb8
2015-12-16 17:32:22,402 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb8 : Unable to read additional data from server sessionid 0x151705ac5a7bdb8, likely server has closed socket
2015-12-16 17:32:22,402 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb8 closed
2015-12-16 17:32:22,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 17185201ns
2015-12-16 17:32:22,402 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:22,402 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:22,403 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:22,403 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:23,388 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:23,389 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:23,403 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:23,403 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:23,404 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:23,405 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:23,406 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:23,408 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b64, negotiated timeout = 5000
2015-12-16 17:32:23,409 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:23,409 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:23,414 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:23,415 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:23,417 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:23,423 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:23,423 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:23,424 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:23,424 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:23,424 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:23,424 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b64
2015-12-16 17:32:23,425 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b64
2015-12-16 17:32:23,425 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b64, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265412859,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:23,426 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b64, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265412859,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:23,427 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b64, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265412860,0 request:: null response:: null
2015-12-16 17:32:23,428 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b64
2015-12-16 17:32:23,428 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b64 : Unable to read additional data from server sessionid 0x251705aced21b64, likely server has closed socket
2015-12-16 17:32:23,428 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b64 closed
2015-12-16 17:32:23,428 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 3721184ns
2015-12-16 17:32:23,428 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:23,428 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:23,430 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:23,430 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #29, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 29, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:23,430 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:23,431 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@24359800 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800)), so the following Deferred is getting paused: Deferred@17538685(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:23,431 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:23,431 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:23,431 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:23,431 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:23,431 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:23,432 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:23,432 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:23,432 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:23,432 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=29, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=5, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:23,433 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:23,433 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:23,433 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:23,434 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,434 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,434 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,435 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,435 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,435 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b65, negotiated timeout = 5000
2015-12-16 17:32:23,436 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,436 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,437 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,437 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:23,437 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:23,439 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:23,440 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,440 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,441 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,441 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,442 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,442 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,443 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,443 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,443 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:23,445 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:23,445 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,446 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,446 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,447 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,447 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,447 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:23,449 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b65, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265412876,0 request:: null response:: null
2015-12-16 17:32:23,449 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b65
2015-12-16 17:32:23,449 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b65 : Unable to read additional data from server sessionid 0x251705aced21b65, likely server has closed socket
2015-12-16 17:32:23,450 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b65 closed
2015-12-16 17:32:23,450 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 18536638ns
2015-12-16 17:32:23,450 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:23,450 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:23,450 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:23,450 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:24,468 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:24,469 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:24,472 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:24,473 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:24,473 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:24,474 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:24,475 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:24,483 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:24,484 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdb9, negotiated timeout = 5000
2015-12-16 17:32:24,484 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:24,484 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:24,484 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:24,484 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:24,485 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:24,487 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,487 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,489 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,489 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,490 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:24,491 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:24,492 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,493 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,493 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:24,493 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:24,493 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,493 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdb9
2015-12-16 17:32:24,494 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdb9
2015-12-16 17:32:24,494 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,494 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,495 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,495 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,496 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,496 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,497 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,497 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:24,498 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,499 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb9, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265413427,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:24,500 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb9, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265413427,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:24,501 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdb9, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265413433,0 request:: null response:: null
2015-12-16 17:32:24,501 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdb9
2015-12-16 17:32:24,501 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdb9 : Unable to read additional data from server sessionid 0x151705ac5a7bdb9, likely server has closed socket
2015-12-16 17:32:24,501 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdb9 closed
2015-12-16 17:32:24,501 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 8187806ns
2015-12-16 17:32:24,502 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:24,502 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:24,510 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Resolved IP of `x.x.x.x' to 10.192.210.33 in 676752ns
2015-12-16 17:32:24,510 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:24,518 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #30, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 30, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:24,518 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@32885422 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422)), so the following Deferred is getting paused: Deferred@30238864(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:24,518 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:24,518 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:24,518 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:24,518 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:24,519 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:24,519 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:24,519 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:24,520 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:24,520 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:24,520 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=30, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=6, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:24,520 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:24,521 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:24,521 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:24,521 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,522 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,522 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,522 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,523 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,523 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,524 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdba, negotiated timeout = 5000
2015-12-16 17:32:24,524 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:24,524 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,524 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,525 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,525 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:24,525 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:24,527 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,528 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,528 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,529 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,529 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,529 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,530 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,530 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,530 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:24,543 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:24,544 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,544 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,545 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,545 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,545 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,546 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:24,548 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdba, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265413462,0 request:: null response:: null
2015-12-16 17:32:24,548 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdba
2015-12-16 17:32:24,548 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdba : Unable to read additional data from server sessionid 0x151705ac5a7bdba, likely server has closed socket
2015-12-16 17:32:24,548 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdba closed
2015-12-16 17:32:24,549 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 30035207ns
2015-12-16 17:32:24,549 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:24,549 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:24,549 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:24,549 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:25,568 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=5, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:25,569 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:25,570 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:25,571 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:25,571 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:25,572 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:25,572 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:25,573 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:25,575 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdbb, negotiated timeout = 5000
2015-12-16 17:32:25,575 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:25,575 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:25,575 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:25,576 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:25,576 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:25,578 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,579 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,580 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,580 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,581 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:25,582 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:25,583 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,583 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,583 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:25,583 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:25,584 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,584 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdbb
2015-12-16 17:32:25,584 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdbb
2015-12-16 17:32:25,584 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,585 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,585 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,586 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,586 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,586 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,587 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,587 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:25,588 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,589 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbb, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265413875,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:25,590 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbb, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265413875,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:25,590 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbb, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265413876,0 request:: null response:: null
2015-12-16 17:32:25,591 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdbb
2015-12-16 17:32:25,591 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdbb : Unable to read additional data from server sessionid 0x151705ac5a7bdbb, likely server has closed socket
2015-12-16 17:32:25,591 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdbb closed
2015-12-16 17:32:25,591 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 7398338ns
2015-12-16 17:32:25,591 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:25,591 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:25,592 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:25,593 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #31, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 31, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:25,593 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:25,593 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:25,593 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@12612745 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745)), so the following Deferred is getting paused: Deferred@19059682(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:25,593 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:25,593 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:25,594 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:25,594 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:25,594 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:25,594 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:25,594 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:25,595 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:25,595 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=31, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=7, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:25,595 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:25,595 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:25,595 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,596 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,596 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,597 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,597 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,597 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,598 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b66, negotiated timeout = 5000
2015-12-16 17:32:25,598 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:25,598 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,599 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,599 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,599 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:25,599 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:25,602 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,602 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,603 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,603 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,604 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,604 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,604 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,605 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,605 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:25,606 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:25,607 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,607 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,608 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,608 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,609 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,609 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:25,611 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b66, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265413887,0 request:: null response:: null
2015-12-16 17:32:25,611 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b66
2015-12-16 17:32:25,611 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b66 : Unable to read additional data from server sessionid 0x251705aced21b66, likely server has closed socket
2015-12-16 17:32:25,611 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b66 closed
2015-12-16 17:32:25,612 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 17890509ns
2015-12-16 17:32:25,612 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:25,612 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:25,612 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:25,612 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:26,669 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=6, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:26,669 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:26,684 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:26,684 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:26,685 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:26,686 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:26,686 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:26,689 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdbc, negotiated timeout = 5000
2015-12-16 17:32:26,690 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:26,690 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:26,694 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:26,694 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:26,694 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:26,695 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:26,695 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:26,696 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:26,696 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,697 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,697 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:26,697 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:26,697 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,697 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdbc
2015-12-16 17:32:26,698 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdbc
2015-12-16 17:32:26,698 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,698 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,699 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,699 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,699 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,700 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,700 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,700 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,701 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,701 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,702 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbc, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265414413,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:26,703 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbc, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265414413,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:26,704 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbc, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265414414,0 request:: null response:: null
2015-12-16 17:32:26,704 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdbc
2015-12-16 17:32:26,704 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdbc : Unable to read additional data from server sessionid 0x151705ac5a7bdbc, likely server has closed socket
2015-12-16 17:32:26,704 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdbc closed
2015-12-16 17:32:26,704 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 6768886ns
2015-12-16 17:32:26,704 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:26,705 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:26,706 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:26,707 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:26,707 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:26,708 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:26,708 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:26,711 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b67, negotiated timeout = 5000
2015-12-16 17:32:26,711 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:26,711 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:26,714 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:26,720 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:26,720 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #32, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 32, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:26,720 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:26,721 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@6036441 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441)), so the following Deferred is getting paused: Deferred@19429653(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:26,721 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:26,721 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:26,722 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b67
2015-12-16 17:32:26,722 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b67
2015-12-16 17:32:26,723 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:26,724 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:26,725 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:26,725 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=32, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=8, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:26,725 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:26,730 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,731 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,731 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,742 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:26,744 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:26,744 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,745 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,745 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,745 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:26,745 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,746 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,746 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,746 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,746 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,747 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,747 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,747 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,748 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,748 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,748 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:26,748 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:26,750 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b67, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265414421,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:26,750 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b67, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265414421,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:26,751 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b67, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265414422,0 request:: null response:: null
2015-12-16 17:32:26,751 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b67
2015-12-16 17:32:26,751 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b67 closed
2015-12-16 17:32:26,751 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 29835743ns
2015-12-16 17:32:26,752 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:26,752 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:26,752 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:27,869 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=7, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:27,869 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:27,882 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:27,883 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:27,884 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:27,885 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:27,885 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:27,886 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:27,888 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec249, negotiated timeout = 5000
2015-12-16 17:32:27,888 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:27,888 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:27,889 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:27,889 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:27,889 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:27,892 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,892 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,893 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,894 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,894 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:27,907 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:27,908 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,909 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,909 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:27,909 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,909 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:27,910 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec249
2015-12-16 17:32:27,910 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,910 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec249
2015-12-16 17:32:27,910 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,911 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,911 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,912 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,912 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:27,913 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,913 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,913 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:27,923 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec249, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265414907,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:27,924 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec249, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265414907,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:27,924 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec249, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265414908,0 request:: null response:: null
2015-12-16 17:32:27,924 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec249
2015-12-16 17:32:27,925 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec249 closed
2015-12-16 17:32:27,925 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 15272420ns
2015-12-16 17:32:27,925 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:27,925 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:27,943 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:27,944 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:27,945 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:27,945 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:27,946 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:27,947 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:27,948 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec24a, negotiated timeout = 5000
2015-12-16 17:32:27,948 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #33, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 33, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:27,948 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:27,948 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:27,949 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@28516270 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441) -> (continuation of Deferred@15011775 after retry RPC@28516270), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441) -> (continuation of Deferred@15011775 after retry RPC@28516270)), so the following Deferred is getting paused: Deferred@15011775(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:27,949 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:27,949 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:27,949 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:27,949 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:27,949 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:27,950 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec24a
2015-12-16 17:32:27,950 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:27,950 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec24a
2015-12-16 17:32:27,950 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:27,951 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=33, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=9, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:27,951 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:27,952 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,953 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,953 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:27,954 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,954 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,955 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:27,955 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:27,957 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:27,958 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,958 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,958 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:27,965 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:27,966 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,966 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,966 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:27,967 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,967 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:27,967 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:27,969 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24a, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265414948,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:27,969 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24a, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265414948,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:27,970 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24a, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265414949,0 request:: null response:: null
2015-12-16 17:32:27,970 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec24a
2015-12-16 17:32:27,970 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec24a : Unable to read additional data from server sessionid 0x351705ac89ec24a, likely server has closed socket
2015-12-16 17:32:27,970 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec24a closed
2015-12-16 17:32:27,970 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 20596478ns
2015-12-16 17:32:27,970 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:27,970 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:27,971 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:29,209 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=8, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:29,209 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:29,222 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:29,223 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:29,223 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:29,225 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:29,226 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:29,226 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:29,228 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec24b, negotiated timeout = 5000
2015-12-16 17:32:29,229 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:29,229 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:29,229 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:29,229 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:29,229 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:29,231 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,232 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,233 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,233 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,233 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:29,235 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:29,235 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,236 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,236 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,236 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,237 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,237 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,237 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:29,237 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,238 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:29,238 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec24b
2015-12-16 17:32:29,238 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,238 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec24b
2015-12-16 17:32:29,238 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,239 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:29,240 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24b, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265415467,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:29,240 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24b, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265415467,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:29,241 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24b, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265415468,0 request:: null response:: null
2015-12-16 17:32:29,241 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec24b
2015-12-16 17:32:29,241 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec24b : Unable to read additional data from server sessionid 0x351705ac89ec24b, likely server has closed socket
2015-12-16 17:32:29,241 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec24b closed
2015-12-16 17:32:29,241 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 3642504ns
2015-12-16 17:32:29,242 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:29,242 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:29,243 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:29,243 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #34, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 34, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:29,244 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@26844364 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441) -> (continuation of Deferred@15011775 after retry RPC@28516270) -> (continuation of Deferred@340901 after retry RPC@26844364), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441) -> (continuation of Deferred@15011775 after retry RPC@28516270) -> (continuation of Deferred@340901 after retry RPC@26844364)), so the following Deferred is getting paused: Deferred@340901(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:29,244 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:29,244 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:29,244 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:29,244 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:29,245 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:29,245 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:29,245 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=34, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=10, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:29,245 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:29,253 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:29,254 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:29,254 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:29,255 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:29,255 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:29,256 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,256 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,257 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,257 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,257 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,257 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,258 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,258 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,258 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,258 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b68, negotiated timeout = 5000
2015-12-16 17:32:29,259 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:29,259 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,259 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,259 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,260 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:29,260 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:29,262 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,262 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,263 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,263 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,263 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,264 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,264 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,264 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,264 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,265 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,265 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:29,266 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:29,267 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,267 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,267 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,268 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,268 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,268 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:29,271 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b68, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265415471,0 request:: null response:: null
2015-12-16 17:32:29,271 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b68
2015-12-16 17:32:29,271 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b68 closed
2015-12-16 17:32:29,271 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 26982325ns
2015-12-16 17:32:29,271 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:29,271 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:29,272 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:29,272 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:30,769 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=9, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:30,769 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:30,783 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:30,784 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:30,785 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:30,786 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:30,786 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:30,787 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:30,789 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b69, negotiated timeout = 5000
2015-12-16 17:32:30,789 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:30,789 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:30,794 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:30,794 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:30,794 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:30,795 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:30,796 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:30,797 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,798 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,798 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:30,798 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,798 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:30,799 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b69
2015-12-16 17:32:30,799 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b69
2015-12-16 17:32:30,809 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,809 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,810 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,810 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,811 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,811 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,812 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,812 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,813 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:30,813 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,815 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b69, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265415904,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:30,816 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b69, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265415904,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:30,817 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b69, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265415905,0 request:: null response:: null
2015-12-16 17:32:30,817 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b69
2015-12-16 17:32:30,817 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b69 : Unable to read additional data from server sessionid 0x251705aced21b69, likely server has closed socket
2015-12-16 17:32:30,817 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b69 closed
2015-12-16 17:32:30,818 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 18909672ns
2015-12-16 17:32:30,818 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:30,818 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:30,831 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:30,840 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:30,840 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #35, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 35, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:30,840 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:30,841 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@8777084 returned Deferred@9156450(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441) -> (continuation of Deferred@15011775 after retry RPC@28516270) -> (continuation of Deferred@340901 after retry RPC@26844364) -> (continuation of Deferred@11183016 after retry RPC@8777084), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@11831907 after retry RPC@32251291) -> (continuation of Deferred@6620237 after retry RPC@30875582) -> (continuation of Deferred@31587399 after retry RPC@26009708) -> (continuation of Deferred@17538685 after retry RPC@24359800) -> (continuation of Deferred@30238864 after retry RPC@32885422) -> (continuation of Deferred@19059682 after retry RPC@12612745) -> (continuation of Deferred@19429653 after retry RPC@6036441) -> (continuation of Deferred@15011775 after retry RPC@28516270) -> (continuation of Deferred@340901 after retry RPC@26844364) -> (continuation of Deferred@11183016 after retry RPC@8777084)), so the following Deferred is getting paused: Deferred@11183016(state=PAUSED, result=Deferred@9156450, callback=<none>, errback=<none>)
2015-12-16 17:32:30,841 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:30,841 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:30,842 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:30,842 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:30,842 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:30,842 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:30,843 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:30,843 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:30,844 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:30,844 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,845 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,845 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,845 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,846 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,846 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdbd, negotiated timeout = 5000
2015-12-16 17:32:30,846 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:30,844 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=35, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:30,847 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,852 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:30,852 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,852 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,853 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:30,853 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:30,856 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,856 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,856 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,857 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,857 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,858 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,858 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,858 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,859 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:30,860 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:30,861 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,861 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,862 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,862 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,864 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,865 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:30,871 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbd, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265415923,0 request:: null response:: null
2015-12-16 17:32:30,872 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdbd
2015-12-16 17:32:30,872 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdbd : Unable to read additional data from server sessionid 0x151705ac5a7bdbd, likely server has closed socket
2015-12-16 17:32:30,872 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdbd closed
2015-12-16 17:32:30,873 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 30558447ns
2015-12-16 17:32:30,873 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:30,873 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:30,873 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:30,874 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:32,888 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=10, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:32,889 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:32,890 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:32,891 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:32,891 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:32,892 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:32,893 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:32,893 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:32,896 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdbe, negotiated timeout = 5000
2015-12-16 17:32:32,897 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:32,897 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:32,897 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:32,897 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:32,897 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:32,899 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,899 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,901 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,901 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,901 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:32,903 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:32,903 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,903 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,904 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:32,904 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,904 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:32,904 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdbe
2015-12-16 17:32:32,904 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,904 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdbe
2015-12-16 17:32:32,904 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,905 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,905 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,906 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,906 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,906 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,907 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:32,907 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,909 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbe, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265416501,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:32,909 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbe, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265416501,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:32,910 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbe, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265416502,0 request:: null response:: null
2015-12-16 17:32:32,910 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdbe
2015-12-16 17:32:32,910 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdbe : Unable to read additional data from server sessionid 0x151705ac5a7bdbe, likely server has closed socket
2015-12-16 17:32:32,910 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdbe closed
2015-12-16 17:32:32,910 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 6389523ns
2015-12-16 17:32:32,911 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:32,911 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:32,912 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:32,912 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:32,913 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Probe Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) failed
org.hbase.async.NonRecoverableException: Too many attempts: Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3632) [asynchbase-1.7.1-20151004.015637-1.jar:na]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) [zookeeper_client-3.4.5.jar:3.4.5-1461890]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [zookeeper_client-3.4.5.jar:3.4.5-1461890]
2015-12-16 17:32:32,913 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:32,913 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Retrying 3 RPCs on NSREd region "hbase:meta,,1"
2015-12-16 17:32:32,913 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #36, payload=BigEndianHeapChannelBuffer(ridx=12, widx=159, cap=159) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x8F\t\x08$\x1A\x03Get \x01\x83\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12n\nbperf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:32:32,914 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:32,914 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #37, payload=BigEndianHeapChannelBuffer(ridx=12, widx=163, cap=163) "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x93\t\x08%\x1A\x03Get \x01\x87\x01\n\x11\x08\x01\x12\x0Dhbase:meta,,1\x12r\nfperf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:\x12\x06\n\x04infoX\x01"
2015-12-16 17:32:32,914 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 147
2015-12-16 17:32:32,914 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:32,914 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 151
2015-12-16 17:32:32,914 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:32,914 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:32,915 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:32,915 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:32,915 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:32,915 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:32,915 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=36, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=3, timeout=0, hasTimedout=false)
2015-12-16 17:32:32,916 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=1) seems to be splitting or closing it.
2015-12-16 17:32:32,916 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:32,916 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:32,917 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=37, response size=932 bytes, 0 readable bytes left, rpc=HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key="perf_namespace:test_opentsdb_uid,\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:AsyncHBase~probe~<;_<,:", region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=3, timeout=0, hasTimedout=false)
2015-12-16 17:32:32,918 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdbf, negotiated timeout = 5000
2015-12-16 17:32:32,918 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:32,918 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,918 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,919 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:32,919 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:32,923 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,924 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:32,925 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:32,925 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,925 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,926 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,926 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,926 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,926 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,927 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,927 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,927 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,927 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,928 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:32,929 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdbf, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265416506,0 request:: null response:: null
2015-12-16 17:32:32,930 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdbf
2015-12-16 17:32:32,930 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdbf closed
2015-12-16 17:32:32,930 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdbf : Unable to read additional data from server sessionid 0x151705ac5a7bdbf, likely server has closed socket
2015-12-16 17:32:32,930 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 15952156ns
2015-12-16 17:32:32,930 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:32,930 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:32,931 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:32,931 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:33,329 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=null)
2015-12-16 17:32:33,329 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:33,343 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:33,344 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:33,344 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:33,345 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:33,345 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:33,348 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec24c, negotiated timeout = 5000
2015-12-16 17:32:33,348 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:33,348 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:33,352 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:33,353 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:33,353 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:33,353 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:33,353 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:33,354 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:33,355 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:33,355 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:33,356 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:33,356 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:33,357 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:33,358 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:33,359 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24c, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265416584,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:33,360 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:33,360 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24c, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265416584,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:33,360 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:33,361 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec24c
2015-12-16 17:32:33,361 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec24c
2015-12-16 17:32:33,372 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24c, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265416585,0 request:: null response:: null
2015-12-16 17:32:33,373 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec24c
2015-12-16 17:32:33,373 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec24c : Unable to read additional data from server sessionid 0x351705ac89ec24c, likely server has closed socket
2015-12-16 17:32:33,373 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec24c closed
2015-12-16 17:32:33,374 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 13187623ns
2015-12-16 17:32:33,374 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:33,374 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:33,388 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:33,388 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:33,389 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:33,390 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:33,390 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:33,393 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b6a, negotiated timeout = 5000
2015-12-16 17:32:33,393 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:33,393 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:33,395 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:33,395 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:33,395 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #38, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 38, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:33,395 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:33,395 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@3078809 returned Deferred@4622398(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809)), so the following Deferred is getting paused: Deferred@10146272(state=PAUSED, result=Deferred@4622398, callback=<none>, errback=<none>)
2015-12-16 17:32:33,396 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:33,397 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:33,397 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:33,397 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=38, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:33,398 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:33,400 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:33,401 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:33,407 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:33,407 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:33,407 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b6a
2015-12-16 17:32:33,407 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:33,407 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b6a
2015-12-16 17:32:33,408 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6a, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265416600,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:33,408 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6a, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265416600,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:33,410 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6a, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265416601,0 request:: null response:: null
2015-12-16 17:32:33,410 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b6a
2015-12-16 17:32:33,410 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b6a : Unable to read additional data from server sessionid 0x251705aced21b6a, likely server has closed socket
2015-12-16 17:32:33,410 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b6a closed
2015-12-16 17:32:33,410 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 3624226ns
2015-12-16 17:32:33,411 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:33,411 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:33,411 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:34,008 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=1, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:34,009 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:34,022 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:34,023 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:34,023 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:34,024 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:34,024 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:34,027 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b6b, negotiated timeout = 5000
2015-12-16 17:32:34,027 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:34,027 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:34,027 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:34,032 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:34,033 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:34,037 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,037 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:34,038 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,038 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:34,038 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:34,038 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b6b
2015-12-16 17:32:34,038 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b6b
2015-12-16 17:32:34,039 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6b, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265416747,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:34,039 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6b, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265416747,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:34,041 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6b, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265416748,0 request:: null response:: null
2015-12-16 17:32:34,041 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b6b
2015-12-16 17:32:34,041 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b6b : Unable to read additional data from server sessionid 0x251705aced21b6b, likely server has closed socket
2015-12-16 17:32:34,041 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b6b closed
2015-12-16 17:32:34,041 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 3314994ns
2015-12-16 17:32:34,042 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:34,042 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:34,060 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:34,061 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=c.c.c.c
2015-12-16 17:32:34,061 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Opening socket connection to server c.c.c.c/3.3.3.3:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:34,062 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Socket connection established to c.c.c.c/3.3.3.3:50512, initiating session
2015-12-16 17:32:34,062 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment request sent on c.c.c.c/3.3.3.3:50512
2015-12-16 17:32:34,064 INFO [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Session establishment complete on server c.c.c.c/3.3.3.3:50512, sessionid = 0x351705ac89ec24d, negotiated timeout = 5000
2015-12-16 17:32:34,065 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:34,065 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:34,068 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:34,070 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:34,071 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,071 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:34,071 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:34,072 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,072 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #39, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 39, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:34,072 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:34,072 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:34,072 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@5123345 returned Deferred@4622398(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345)), so the following Deferred is getting paused: Deferred@7401649(state=PAUSED, result=Deferred@4622398, callback=<none>, errback=<none>)
2015-12-16 17:32:34,072 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x351705ac89ec24d
2015-12-16 17:32:34,072 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:34,073 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x351705ac89ec24d
2015-12-16 17:32:34,073 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24d, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265416751,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:34,073 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:34,074 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:34,074 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24d, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265416751,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:34,074 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=39, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:34,074 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:34,076 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: Reading reply sessionid:0x351705ac89ec24d, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265416752,0 request:: null response:: null
2015-12-16 17:32:34,076 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x351705ac89ec24d
2015-12-16 17:32:34,076 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(c.c.c.c:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x351705ac89ec24d : Unable to read additional data from server sessionid 0x351705ac89ec24d, likely server has closed socket
2015-12-16 17:32:34,076 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x351705ac89ec24d closed
2015-12-16 17:32:34,076 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 3865311ns
2015-12-16 17:32:34,077 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:34,077 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:34,082 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:34,889 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=2, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:34,889 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:34,905 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:34,906 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=a.a.a.a
2015-12-16 17:32:34,906 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Opening socket connection to server a.a.a.a/1.1.1.1:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:34,907 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Socket connection established to a.a.a.a/1.1.1.1:50512, initiating session
2015-12-16 17:32:34,907 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment request sent on a.a.a.a/1.1.1.1:50512
2015-12-16 17:32:34,910 INFO [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Session establishment complete on server a.a.a.a/1.1.1.1:50512, sessionid = 0x151705ac5a7bdc0, negotiated timeout = 5000
2015-12-16 17:32:34,910 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:34,910 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:34,914 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:34,916 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:34,918 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:34,918 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,919 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:34,919 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,919 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:34,919 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:34,919 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x151705ac5a7bdc0
2015-12-16 17:32:34,919 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x151705ac5a7bdc0
2015-12-16 17:32:34,920 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdc0, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265416903,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:34,920 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdc0, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265416903,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:34,922 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: Reading reply sessionid:0x151705ac5a7bdc0, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265416904,0 request:: null response:: null
2015-12-16 17:32:34,922 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x151705ac5a7bdc0
2015-12-16 17:32:34,922 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(a.a.a.a:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x151705ac5a7bdc0 : Unable to read additional data from server sessionid 0x151705ac5a7bdc0, likely server has closed socket
2015-12-16 17:32:34,922 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x151705ac5a7bdc0 closed
2015-12-16 17:32:34,922 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 3089707ns
2015-12-16 17:32:34,923 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:34,923 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:34,924 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:34,924 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #40, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 40, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:34,924 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:34,925 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@26410710 returned Deferred@4622398(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345) -> (continuation of Deferred@26790598 after retry RPC@26410710), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345) -> (continuation of Deferred@26790598 after retry RPC@26410710)), so the following Deferred is getting paused: Deferred@26790598(state=PAUSED, result=Deferred@4622398, callback=<none>, errback=<none>)
2015-12-16 17:32:34,925 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:34,925 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:34,925 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:34,925 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:34,926 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:34,926 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:34,926 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:34,926 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:34,927 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=40, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:34,927 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:34,927 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:34,927 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:34,928 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,928 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,928 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,929 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,929 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,929 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,930 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b6c, negotiated timeout = 5000
2015-12-16 17:32:34,930 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,930 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,940 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,940 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,940 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:34,940 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:34,943 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,943 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,943 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,944 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,944 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,944 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,945 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,945 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,945 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:34,947 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:34,947 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,948 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,948 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,948 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,949 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,949 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:34,960 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6c, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265416912,0 request:: null response:: null
2015-12-16 17:32:34,961 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b6c
2015-12-16 17:32:34,961 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b6c : Unable to read additional data from server sessionid 0x251705aced21b6c, likely server has closed socket
2015-12-16 17:32:34,961 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b6c closed
2015-12-16 17:32:34,962 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 35993002ns
2015-12-16 17:32:34,962 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:34,962 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:34,962 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:34,962 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:35,929 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=3, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:35,929 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:35,930 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:35,931 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:35,931 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:35,932 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:35,933 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:35,933 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:35,936 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b6d, negotiated timeout = 5000
2015-12-16 17:32:35,936 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:35,936 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:35,936 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:35,936 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:35,936 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:35,938 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,938 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,940 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,941 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,941 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:35,942 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:35,942 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,942 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,943 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:35,943 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,943 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:35,943 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b6d
2015-12-16 17:32:35,943 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,943 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b6d
2015-12-16 17:32:35,944 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,944 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,944 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,945 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,945 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,945 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,945 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:35,946 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,947 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6d, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265417195,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:35,947 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6d, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265417195,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:35,948 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6d, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265417196,0 request:: null response:: null
2015-12-16 17:32:35,948 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b6d
2015-12-16 17:32:35,948 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b6d : Unable to read additional data from server sessionid 0x251705aced21b6d, likely server has closed socket
2015-12-16 17:32:35,948 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b6d closed
2015-12-16 17:32:35,948 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 4992883ns
2015-12-16 17:32:35,948 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:35,949 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:35,949 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:35,950 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #41, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 41, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:35,950 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@21208814 returned Deferred@4622398(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345) -> (continuation of Deferred@26790598 after retry RPC@26410710) -> (continuation of Deferred@7806978 after retry RPC@21208814), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345) -> (continuation of Deferred@26790598 after retry RPC@26410710) -> (continuation of Deferred@7806978 after retry RPC@21208814)), so the following Deferred is getting paused: Deferred@7806978(state=PAUSED, result=Deferred@4622398, callback=<none>, errback=<none>)
2015-12-16 17:32:35,950 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:35,950 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:35,950 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:35,951 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:35,951 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:35,951 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:35,951 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:35,951 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:35,952 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:35,952 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=41, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=5, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:35,952 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:35,952 INFO [AsyncHBase I/O Worker #1] HBaseClient: Invalidated cache for .META. as RegionClient@16233423(chan=[id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) still seems to be splitting or closing it.
2015-12-16 17:32:35,952 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:35,953 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,953 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,953 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,954 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,954 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,954 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,954 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,955 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b6e, negotiated timeout = 5000
2015-12-16 17:32:35,955 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:35,955 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,955 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,955 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,956 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:35,956 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:35,957 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,958 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,958 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,958 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,959 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,959 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,959 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,959 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,959 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,960 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,960 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:35,961 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:35,961 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,961 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,961 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,962 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,962 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,962 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,962 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,962 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:35,964 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6e, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,305265417203,0 request:: null response:: null
2015-12-16 17:32:35,964 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b6e
2015-12-16 17:32:35,964 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b6e : Unable to read additional data from server sessionid 0x251705aced21b6e, likely server has closed socket
2015-12-16 17:32:35,964 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b6e closed
2015-12-16 17:32:35,965 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 13758377ns
2015-12-16 17:32:35,965 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:35,965 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:35,965 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:35,965 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down
2015-12-16 17:32:36,988 DEBUG [AsyncHBase Timer HBaseClient #1] HBaseClient: Done waiting after NSRE on "hbase:meta,,1", retrying Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=4, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""))
2015-12-16 17:32:36,989 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:36,990 INFO [AsyncHBase Timer HBaseClient #1] HBaseClient: Need to find the .META. region
2015-12-16 17:32:36,991 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:36,991 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:36,992 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:36,993 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Socket connection established to b.b.b.b/2.2.2.2:50512, initiating session
2015-12-16 17:32:36,993 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment request sent on b.b.b.b/2.2.2.2:50512
2015-12-16 17:32:36,996 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Session establishment complete on server b.b.b.b/2.2.2.2:50512, sessionid = 0x251705aced21b6f, negotiated timeout = 5000
2015-12-16 17:32:36,996 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=0
2015-12-16 17:32:36,996 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:36,996 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=0)
2015-12-16 17:32:36,996 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Finding the -ROOT- or .META. region in ZooKeeper
2015-12-16 17:32:36,996 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2015-12-16 17:32:36,998 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:36,999 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,001 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,001 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,001 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: saslClient.evaluateChallenge(len=50)
2015-12-16 17:32:37,003 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: ClientCnxn:sendSaslPacket:length=73
2015-12-16 17:32:37,003 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,004 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,004 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Got ZooKeeper event: WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:37,004 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,004 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:37,005 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x251705aced21b6f
2015-12-16 17:32:37,005 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,005 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x251705aced21b6f
2015-12-16 17:32:37,005 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,005 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,006 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,006 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,006 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:37,007 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/root-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,007 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 0,4 replyHeader:: 0,0,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: until SASL authentication completes.
2015-12-16 17:32:37,007 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxnSocketNIO: deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null response:: nulluntil SASL authentication completes.
2015-12-16 17:32:37,009 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6f, packet:: clientPath:/hbase/reluxred-hb1/root-region-server serverPath:/hbase/reluxred-hb1/root-region-server finished:false header:: 3,4 replyHeader:: 3,305265417677,-101 request:: '/hbase/reluxred-hb1/root-region-server,T response::
2015-12-16 17:32:37,010 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6f, packet:: clientPath:/hbase/reluxred-hb1/meta-region-server serverPath:/hbase/reluxred-hb1/meta-region-server finished:false header:: 4,4 replyHeader:: 4,305265417677,0 request:: '/hbase/reluxred-hb1/meta-region-server,T response:: #ffffffff000146d61737465723a35303531306150fffffffc23ffffffc434ffffffe01d50425546a2ba1e677372643234336e33382e7265642e79677269642e7961686f6f2e636f6d10ffffffcfffffff8a318ffffffb1ffffffa8ffffffe0ffffff83ffffff972a100183ffffffa861ffffffb060,s{304942735609,304943234458,1449289192576,1449290927291,9,0,0,0,84,0,304942735609}
2015-12-16 17:32:37,010 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Reading reply sessionid:0x251705aced21b6f, packet:: clientPath:null serverPath:null finished:false header:: 5,-11 replyHeader:: 5,305265417678,0 request:: null response:: null
2015-12-16 17:32:37,010 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Disconnecting client for session: 0x251705aced21b6f
2015-12-16 17:32:37,010 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: An exception was thrown while closing send thread for session 0x251705aced21b6f : Unable to read additional data from server sessionid 0x251705aced21b6f, likely server has closed socket
2015-12-16 17:32:37,010 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x251705aced21b6f closed
2015-12-16 17:32:37,011 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: ZooKeeper#close completed in 6079316ns
2015-12-16 17:32:37,011 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: No longer connected to ZooKeeper, event=WatchedEvent state:SaslAuthenticated type:None path:null
2015-12-16 17:32:37,011 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=a.a.a.a:50512,b.b.b.b:50512,c.c.c.c:50512 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@10a0018
2015-12-16 17:32:37,012 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.192.210.33:50511
2015-12-16 17:32:37,013 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] RegionClient: [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] Sending RPC #42, payload=BigEndianHeapChannelBuffer(ridx=12, widx=217, cap=217) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -55, 9, 8, 42, 26, 3, 71, 101, 116, 32, 1, -67, 1, 10, 17, 8, 1, 18, 13, 104, 98, 97, 115, 101, 58, 109, 101, 116, 97, 44, 44, 49, 18, -89, 1, 10, -94, 1, 112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 80, 1]
2015-12-16 17:32:37,013 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2015-12-16 17:32:37,013 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] Deferred: callback=retry RPC@14553247 returned Deferred@4622398(state=PENDING, result=null, callback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345) -> (continuation of Deferred@26790598 after retry RPC@26410710) -> (continuation of Deferred@7806978 after retry RPC@21208814) -> (continuation of Deferred@18174685 after retry RPC@14553247), errback=retry other RPCs NSRE'd on "hbase:meta,,1" -> (continuation of Deferred@10146272 after retry RPC@3078809) -> (continuation of Deferred@7401649 after retry RPC@5123345) -> (continuation of Deferred@26790598 after retry RPC@26410710) -> (continuation of Deferred@7806978 after retry RPC@21208814) -> (continuation of Deferred@18174685 after retry RPC@14553247)), so the following Deferred is getting paused: Deferred@18174685(state=PAUSED, result=Deferred@4622398, callback=<none>, errback=<none>)
2015-12-16 17:32:37,013 DEBUG [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ZooKeeperSaslClient: creating sasl client: client=foo@grid.bar.com;service=zookeeper;serviceHostname=b.b.b.b
2015-12-16 17:32:37,013 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] WRITTEN_AMOUNT: 205
2015-12-16 17:32:37,014 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Ignore any DEBUG exception from ZooKeeper
2015-12-16 17:32:37,014 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Closing session: 0x0
2015-12-16 17:32:37,014 DEBUG [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: Closing client for session: 0x0
2015-12-16 17:32:37,014 DEBUG [AsyncHBase I/O Worker #1] RegionClient: handleUpstream [id: 0xa87c9ccf, /4.4.4.4:41938 => /10.192.210.33:50511] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=932, cap=932)
2015-12-16 17:32:37,015 DEBUG [AsyncHBase I/O Worker #1] RegionClient: ------------------>> ENTERING DECODE >>------------------
2015-12-16 17:32:37,015 INFO [AsyncHBase Timer HBaseClient #1-SendThread(b.b.b.b:50512)] ClientCnxn: Opening socket connection to server b.b.b.b/2.2.2.2:50512. Will attempt to SASL-authenticate using Login Context section 'Client'
2015-12-16 17:32:37,015 DEBUG [AsyncHBase I/O Worker #1] RegionClient: rpcid=42, response size=932 bytes, 0 readable bytes left, rpc=Exists(table="hbase:meta", key=[112, 101, 114, 102, 95, 110, 97, 109, 101, 115, 112, 97, 99, 101, 58, 102, 108, 105, 99, 107, 114, 95, 111, 112, 101, 110, 116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment