Skip to content

Instantly share code, notes, and snippets.

@justenwalker
Created January 9, 2014 17:33

Revisions

  1. justenwalker created this gist Jan 9, 2014.
    50 changes: 50 additions & 0 deletions unsafeAssassinateEndpoint.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,50 @@
    # Unsafe Node Assassinate

    On Cassandra 1.1.x, nodetool removetoken <token> on a dead node hangs. The only way to reliably remove the node
    from the cluster is to use an undocumented JMX command to force the remove of the node from gossip.

    Upgrading to 1.2.x and above should obviate these steps in favor of `nodetool removenode` - [See 1.2 Docs](http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/tools/toolsNodetool_r.html#reference_ds_f4n_y2j_gk__removenode)

    It's probably also a good idea to run a `nodetool repair` after this operation - [See wiki](http://wiki.apache.org/cassandra/Operations#Replacing_a_Dead_Node)

    ## Prerequisites

    ### 1. Log onto existing cluster alive node
    ### 2. Download [JMX Term](http://wiki.cyclopsgroup.org/jmxterm/download)

    #### wget
    ```
    $ wget -q -O jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar
    ```
    #### curl
    ```
    $ curl -s -o jmxterm.jar http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar
    ```

    ### 3. Run jmxterm
    ```
    $ java -jar ./jmxterm.jar
    Welcome to JMX terminal. Type "help" for available commands.
    $>
    ```

    ## Assassinate node

    Example bad node: `10.0.0.100`

    - Connect to the local cluster
    - Select the Gossiper MBean
    - Run the unsafeAssassinateEndpoint with the ip of the bad node

    ```
    $>open localhost:7199
    #Connection to localhost:7199 is opened
    $>bean org.apache.cassandra.net:type=Gossiper
    #bean is set to org.apache.cassandra.net:type=Gossiper
    $>run unsafeAssassinateEndpoint 10.0.0.100
    #calling operation unsafeAssassinateEndpoint of mbean org.apache.cassandra.net:type=Gossiper
    #operation returns:
    null
    $>quit
    #bye
    ```