scooby doo and the case of the phantom hbase regions

gistfile1.txt

hbck has been emitting complaints about regions that are neither in HDFS or in .META.:

14/01/06 21:21:12 WARN util.HBaseFsck: No HDFS region dir found: { meta => null, hdfs => null, deployed => hbase3-rs12.klout,60020,1388535340339;moments_time_20131229,5000,1388306209624.bf91db8893ce885b5ac0cc472cc5e661. } meta=null
and
 ERROR: No regioninfo in Meta or HDFS. { meta => null, hdfs => null, deployed => hbase3-rs12.klout,60020,1388535340339;moments_time_20131229,5000,1388306209624.bf91db8893ce885b5ac0cc472cc5e661. }

After the region server crashes last week, this was logged:

ABORTING region server hbase3-rs24.klout,60020,1386199155964: Exception refreshing OPENING; region=bf91db8893ce885b5ac0cc472cc5e661, context=post_open_deploy
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/unassigned/bf91db8893ce885b5ac0cc472cc5e661
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:290)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:746)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:767)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:747)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.retransitionNodeOpening(ZKAssign.java:674)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.tickleOpening(OpenRegionHandler.java:375)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.updateMeta(OpenRegionHandler.java:180)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        
Turns out there were region servers that were in a bad state, shutting down the hmaster and bouncing all of the regionservers that had "phantom" regions brought the cluster back to consistency again.