Skip to content

Instantly share code, notes, and snippets.

@angrycub angrycub/process.md
Last active Oct 17, 2017

Embed
What would you like to do?
Using Force Remove to Change a nodes IP address.

##Using Force Replace to Change a nodes IP address. ###Objective Previous to Riak 1.2, a cluster node's IP address could be changed by running riak-admin reip on each cluster node. As of Riak 1.2, this method has been replaced with riak-admin cluster force-replace which is safer and does not require any downtime.

###Scenario

Riak is running in a cluster of five nodes.

  • riak@64.214.83.25 on node1.localdomain (192.168.17.11)(64.214.83.25)
  • riak@64.214.83.26 on node2.localdomain (192.168.17.12)(64.214.83.26)
  • riak@64.214.83.27 on node3.localdomain (192.168.17.13)(64.214.83.27)
  • riak@64.214.83.28 on node4.localdomain (192.168.17.14)(64.214.83.28)
  • riak@64.214.83.29 on node5.localdomain (192.168.17.15)(64.214.83.29)

The cluster is currently using the public IPs for communication.

The goal is to change the cluster to work over the private IP addresses with no downtime.

The Process tl;dr

This process can be accomplished in three steps, the details of which will be discussed below.


The Process

Down the Node

  1. Stop riak on node1.localdomain if the node is still running in any way.

    riak stop

    node1> riak stop
    Attempting to restart script through sudo -H -u riak
    ok
    node1>
    
  2. Mark riak@64.214.83.25 down from node2.localdomain

riak-admin down riak@64.214.83.25

```
node2> riak-admin down riak@64.214.83.25
Attempting to restart script through sudo -H -u riak
Success: "riak@64.214.83.25" marked as down
node2>
```
This will tell the cluster that this node is offline and ring-state transtions should be allowed, and can be run from any running cluster node.

Reconfigure the node to Listen on the Private IP

Reconfigure node1.localdomain to listen on the private IP addresses by doing the following:

  1. Edit the vm.args file on the new node and set the -name argument as follows:

    -name riak@192.168.17.11
    
  2. Change the IP addresses in app.config as appropriate, specifically pb_ip,http,https, and cluster_mgr settings. Verify that they are listening on the appropriate addresses. These are generally load-balancer or public facing IP addresses.

  3. Rename the ring folder. The location of the ring folder is listed in the /etc/riak/app.config.

  4. Start Riak on node1.localdomain.

    riak start

    node1> riak start
    Attempting to restart script through sudo -H -u riak    
    node1>
    
  5. Join the node back into the cluster.

    riak-admin cluster join riak@64.214.83.26

    node1> riak-admin cluster join riak@64.214.83.26
    Attempting to restart script through sudo -H -u riak
    Success: staged join request for 'riak@192.168.17.11' to 'riak@64.214.83.26'
    node1>
    
  6. Use force-replace to change all ownership references from riak@64.214.83.25 to riak@192.168.17.11.

riak-admin cluster force-replace riak@64.214.83.25 riak@192.168.17.11

```
node1> riak-admin cluster force-replace riak@64.214.83.25 riak@192.168.17.11
Attempting to restart script through sudo -H -u riak
Success: staged forced replacement of 'riak@64.214.83.25' with 'riak@192.168.17.11'
node1>
```
  1. Show the planned cluster changes.

    riak-admin cluster plan

    node1> riak-admin cluster plan
    Attempting to restart script through sudo -H -u riak
    =========================== Staged Changes ============================
    Action         Nodes(s)
    -----------------------------------------------------------------------
    join           'riak@192.168.17.11'
    force-replace  'riak@64.214.83.25' with 'riak@192.168.17.11'
    -----------------------------------------------------------------------
    
    WARNING: All of 'riak@64.214.83.25' replicas will be lost
    
    NOTE: Applying these changes will result in 1 cluster transition
    
    #######################################################################
                         After cluster transition 1/1
    #######################################################################
    
    ============================= Membership ==============================
    Status     Ring    Pending    Node
    -----------------------------------------------------------------------
    valid      20.3%      --      'riak@192.168.17.11'
    valid      20.3%      --      'riak@64.214.83.26'
    valid      20.3%      --      'riak@64.214.83.27'
    valid      20.3%      --      'riak@64.214.83.28'
    valid      18.8%      --      'riak@64.214.83.29'
    -----------------------------------------------------------------------
    Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
    
    Partitions reassigned from cluster changes: 13
      13 reassigned from 'riak@64.214.83.25' to 'riak@192.168.17.11'
    
    node1>
    
  2. Commit the changes to the cluster.

    riak-admin cluster commit

    node1> riak-admin cluster commit
    Attempting to restart script through sudo -H -u riak
    Cluster changes committed
    node1>
    
  3. Check that everything connected and functioning as expected

    riak-admin member-status

    node1> riak-admin member-status
    Attempting to restart script through sudo -H -u riak
    ============================= Membership ==============================
    Status     Ring    Pending    Node
    -----------------------------------------------------------------------
    valid      20.3%      --      'riak@192.168.17.11'
    valid      20.3%      --      'riak@64.214.83.26'
    valid      20.3%      --      'riak@64.214.83.27'
    valid      20.3%      --      'riak@64.214.83.28'
    valid      18.8%      --      'riak@64.214.83.29'
    -----------------------------------------------------------------------
    Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
    
  4. Monitor riak-admin transfers to make sure that all hinted handoff is complete.

  5. Delete the renamed ring folder to clean up once everything is done.

Repeat

Repeat for the remaining nodes. Use riak@192.168.17.11 for further cluster join commands in Replace:4

riak-admin cluster join riak@192.168.17.11

node2> riak-admin cluster join riak@192.168.17.11
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak@192.168.17.12' to 'riak@192.168.17.11'
node2>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.