angrycub/process.md

## process.md

      
    Raw
  

              process.md
            
          
    ##Using Force Replace to Change a nodes IP address.
###Objective
Previous to Riak 1.2, a cluster node's IP address could be changed by running riak-admin reip on each cluster node.   As of Riak 1.2, this method has been replaced with riak-admin cluster force-replace which is safer and does not require any downtime.
###Scenario
Riak is running in a cluster of five nodes.

riak@64.214.83.25 on node1.localdomain (192.168.17.11)(64.214.83.25)
riak@64.214.83.26 on node2.localdomain (192.168.17.12)(64.214.83.26)
riak@64.214.83.27 on node3.localdomain (192.168.17.13)(64.214.83.27)
riak@64.214.83.28 on node4.localdomain (192.168.17.14)(64.214.83.28)
riak@64.214.83.29 on node5.localdomain (192.168.17.15)(64.214.83.29)

The cluster is currently using the public IPs for communication.
The goal is to change the cluster to work over the private IP addresses with no downtime.
The Process tl;dr

This process can be accomplished in three steps, the details of which will be discussed below.

Down the Node
Reconfigure the node to Listen on the Private IP
Repeat


The Process

Down the Node


Stop riak on node1.localdomain if the node is still running in any way.

riak stop

node1> riak stop
Attempting to restart script through sudo -H -u riak
ok
node1>


Mark riak@64.214.83.25 down from node2.localdomain


riak-admin down riak@64.214.83.25

```
node2> riak-admin down riak@64.214.83.25
Attempting to restart script through sudo -H -u riak
Success: "riak@64.214.83.25" marked as down
node2>
```
This will tell the cluster that this node is offline and ring-state transtions should be allowed, and can be run from any running cluster node.

Reconfigure the node to Listen on the Private IP

Reconfigure node1.localdomain to listen on the private IP addresses by doing the following:


Edit the vm.args file on the new node and set the -name argument as follows:
-name riak@192.168.17.11


Change the IP addresses in app.config as appropriate, specifically pb_ip,http,https, and cluster_mgr settings.  Verify that they are listening on the appropriate addresses.  These are generally load-balancer or public facing IP addresses.


Rename the ring folder.  The location of the ring folder is listed in the /etc/riak/app.config.


Start Riak on node1.localdomain.

riak start

node1> riak start
Attempting to restart script through sudo -H -u riak    
node1>


Join the node back into the cluster. 

riak-admin cluster join riak@64.214.83.26

node1> riak-admin cluster join riak@64.214.83.26
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak@192.168.17.11' to 'riak@64.214.83.26'
node1>


Use force-replace to change all ownership references from riak@64.214.83.25 to riak@192.168.17.11.


riak-admin cluster force-replace riak@64.214.83.25 riak@192.168.17.11

```
node1> riak-admin cluster force-replace riak@64.214.83.25 riak@192.168.17.11
Attempting to restart script through sudo -H -u riak
Success: staged forced replacement of 'riak@64.214.83.25' with 'riak@192.168.17.11'
node1>
```


Show the planned cluster changes.

riak-admin cluster plan

node1> riak-admin cluster plan
Attempting to restart script through sudo -H -u riak
=========================== Staged Changes ============================
Action         Nodes(s)
-----------------------------------------------------------------------
join           'riak@192.168.17.11'
force-replace  'riak@64.214.83.25' with 'riak@192.168.17.11'
-----------------------------------------------------------------------

WARNING: All of 'riak@64.214.83.25' replicas will be lost

NOTE: Applying these changes will result in 1 cluster transition

#######################################################################
                     After cluster transition 1/1
#######################################################################

============================= Membership ==============================
Status     Ring    Pending    Node
-----------------------------------------------------------------------
valid      20.3%      --      'riak@192.168.17.11'
valid      20.3%      --      'riak@64.214.83.26'
valid      20.3%      --      'riak@64.214.83.27'
valid      20.3%      --      'riak@64.214.83.28'
valid      18.8%      --      'riak@64.214.83.29'
-----------------------------------------------------------------------
Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

Partitions reassigned from cluster changes: 13
  13 reassigned from 'riak@64.214.83.25' to 'riak@192.168.17.11'

node1>


Commit the changes to the cluster.

riak-admin cluster commit

node1> riak-admin cluster commit
Attempting to restart script through sudo -H -u riak
Cluster changes committed
node1>


Check that everything connected and functioning as expected

riak-admin member-status

node1> riak-admin member-status
Attempting to restart script through sudo -H -u riak
============================= Membership ==============================
Status     Ring    Pending    Node
-----------------------------------------------------------------------
valid      20.3%      --      'riak@192.168.17.11'
valid      20.3%      --      'riak@64.214.83.26'
valid      20.3%      --      'riak@64.214.83.27'
valid      20.3%      --      'riak@64.214.83.28'
valid      18.8%      --      'riak@64.214.83.29'
-----------------------------------------------------------------------
Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0


Monitor riak-admin transfers to make sure that all hinted handoff is complete.


Delete the renamed ring folder to clean up once everything is done.


Repeat

Repeat for the remaining nodes. Use riak@192.168.17.11 for further cluster join commands in Replace:4
riak-admin cluster join riak@192.168.17.11
node2> riak-admin cluster join riak@192.168.17.11
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak@192.168.17.12' to 'riak@192.168.17.11'
node2>