Cassandra unsafe assassinate - removing a dead node from the cluster

unsafeAssassinateEndpoint.md

Unsafe Node Assassinate

On Cassandra 1.1.x, nodetool removetoken on a dead node hangs. The only way to reliably remove the node from the cluster is to use an undocumented JMX command to force the remove of the node from gossip.

Upgrading to 1.2.x and above should obviate these steps in favor of nodetool removenode - See 1.2 Docs

It's probably also a good idea to run a nodetool repair after this operation - See wiki

Prerequisites

1. Log onto existing cluster alive node

2. Download JMX Term

wget

$ wget -q -O jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar

curl

$ curl -s -o jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar

3. Run jmxterm

$ java -jar ./jmxterm.jar
Welcome to JMX terminal. Type "help" for available commands.
$>

Assassinate node

Example bad node: 10.0.0.100

• Connect to the local cluster
• Select the Gossiper MBean
• Run the unsafeAssassinateEndpoint with the ip of the bad node

$>open localhost:7199
#Connection to localhost:7199 is opened
$>bean org.apache.cassandra.net:type=Gossiper
#bean is set to org.apache.cassandra.net:type=Gossiper
$>run unsafeAssassinateEndpoint 10.0.0.100
#calling operation unsafeAssassinateEndpoint of mbean org.apache.cassandra.net:type=Gossiper
#operation returns:
null
$>quit
#bye