Skip to content

Instantly share code, notes, and snippets.

@justenwalker

justenwalker/unsafeAssassinateEndpoint.md

Created January 9, 2014 17:33
Show Gist options
  • Star 10 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save justenwalker/8338334 to your computer and use it in GitHub Desktop.
Save justenwalker/8338334 to your computer and use it in GitHub Desktop.
Download ZIP
Cassandra unsafe assassinate - removing a dead node from the cluster
Raw
unsafeAssassinateEndpoint.md

Unsafe Node Assassinate

On Cassandra 1.1.x, nodetool removetoken on a dead node hangs. The only way to reliably remove the node from the cluster is to use an undocumented JMX command to force the remove of the node from gossip.

Upgrading to 1.2.x and above should obviate these steps in favor of nodetool removenode - See 1.2 Docs

It's probably also a good idea to run a nodetool repair after this operation - See wiki

Prerequisites

1. Log onto existing cluster alive node

2. Download JMX Term

wget

$ wget -q -O jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar

curl

$ curl -s -o jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar

3. Run jmxterm

$ java -jar ./jmxterm.jar
Welcome to JMX terminal. Type "help" for available commands.
$>

Assassinate node

Example bad node: 10.0.0.100

  • Connect to the local cluster
  • Select the Gossiper MBean
  • Run the unsafeAssassinateEndpoint with the ip of the bad node
$>open localhost:7199
#Connection to localhost:7199 is opened
$>bean org.apache.cassandra.net:type=Gossiper
#bean is set to org.apache.cassandra.net:type=Gossiper
$>run unsafeAssassinateEndpoint 10.0.0.100
#calling operation unsafeAssassinateEndpoint of mbean org.apache.cassandra.net:type=Gossiper
#operation returns:
null
$>quit
#bye
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment