Created
January 9, 2014 17:33
Revisions
-
justenwalker created this gist
Jan 9, 2014 .There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,50 @@ # Unsafe Node Assassinate On Cassandra 1.1.x, nodetool removetoken <token> on a dead node hangs. The only way to reliably remove the node from the cluster is to use an undocumented JMX command to force the remove of the node from gossip. Upgrading to 1.2.x and above should obviate these steps in favor of `nodetool removenode` - [See 1.2 Docs](http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/tools/toolsNodetool_r.html#reference_ds_f4n_y2j_gk__removenode) It's probably also a good idea to run a `nodetool repair` after this operation - [See wiki](http://wiki.apache.org/cassandra/Operations#Replacing_a_Dead_Node) ## Prerequisites ### 1. Log onto existing cluster alive node ### 2. Download [JMX Term](http://wiki.cyclopsgroup.org/jmxterm/download) #### wget ``` $ wget -q -O jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar ``` #### curl ``` $ curl -s -o jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar ``` ### 3. Run jmxterm ``` $ java -jar ./jmxterm.jar Welcome to JMX terminal. Type "help" for available commands. $> ``` ## Assassinate node Example bad node: `10.0.0.100` - Connect to the local cluster - Select the Gossiper MBean - Run the unsafeAssassinateEndpoint with the ip of the bad node ``` $>open localhost:7199 #Connection to localhost:7199 is opened $>bean org.apache.cassandra.net:type=Gossiper #bean is set to org.apache.cassandra.net:type=Gossiper $>run unsafeAssassinateEndpoint 10.0.0.100 #calling operation unsafeAssassinateEndpoint of mbean org.apache.cassandra.net:type=Gossiper #operation returns: null $>quit #bye ```