##Using Force Replace to Change a nodes IP address.
###Objective
Previous to Riak 1.2, a cluster node's IP address could be changed by running riak-admin reip
on each cluster node. As of Riak 1.2, this method has been replaced with riak-admin cluster force-replace
which is safer and does not require any downtime.
###Scenario
Riak is running in a cluster of five nodes.
riak@64.214.83.25
onnode1.localdomain
(192.168.17.11)(64.214.83.25)riak@64.214.83.26
onnode2.localdomain
(192.168.17.12)(64.214.83.26)riak@64.214.83.27
onnode3.localdomain
(192.168.17.13)(64.214.83.27)riak@64.214.83.28
onnode4.localdomain
(192.168.17.14)(64.214.83.28)riak@64.214.83.29
onnode5.localdomain
(192.168.17.15)(64.214.83.29)
The cluster is currently using the public IPs for communication.
The goal is to change the cluster to work over the private IP addresses with no downtime.
This process can be accomplished in three steps, the details of which will be discussed below.
-
Stop riak on
node1.localdomain
if the node is still running in any way.riak stop
node1> riak stop Attempting to restart script through sudo -H -u riak ok node1>
-
Mark
riak@64.214.83.25
down fromnode2.localdomain
riak-admin down riak@64.214.83.25
```
node2> riak-admin down riak@64.214.83.25
Attempting to restart script through sudo -H -u riak
Success: "riak@64.214.83.25" marked as down
node2>
```
This will tell the cluster that this node is offline and ring-state transtions should be allowed, and can be run from any running cluster node.
Reconfigure node1.localdomain
to listen on the private IP addresses by doing the following:
-
Edit the
vm.args
file on the new node and set the-name
argument as follows:-name riak@192.168.17.11
-
Change the IP addresses in
app.config
as appropriate, specificallypb_ip
,http
,https
, andcluster_mgr
settings. Verify that they are listening on the appropriate addresses. These are generally load-balancer or public facing IP addresses. -
Rename the ring folder. The location of the ring folder is listed in the
/etc/riak/app.config
. -
Start Riak on
node1.localdomain
.riak start
node1> riak start Attempting to restart script through sudo -H -u riak node1>
-
Join the node back into the cluster.
riak-admin cluster join riak@64.214.83.26
node1> riak-admin cluster join riak@64.214.83.26 Attempting to restart script through sudo -H -u riak Success: staged join request for 'riak@192.168.17.11' to 'riak@64.214.83.26' node1>
-
Use
force-replace
to change all ownership references fromriak@64.214.83.25
toriak@192.168.17.11
.
riak-admin cluster force-replace riak@64.214.83.25 riak@192.168.17.11
```
node1> riak-admin cluster force-replace riak@64.214.83.25 riak@192.168.17.11
Attempting to restart script through sudo -H -u riak
Success: staged forced replacement of 'riak@64.214.83.25' with 'riak@192.168.17.11'
node1>
```
-
Show the planned cluster changes.
riak-admin cluster plan
node1> riak-admin cluster plan Attempting to restart script through sudo -H -u riak =========================== Staged Changes ============================ Action Nodes(s) ----------------------------------------------------------------------- join 'riak@192.168.17.11' force-replace 'riak@64.214.83.25' with 'riak@192.168.17.11' ----------------------------------------------------------------------- WARNING: All of 'riak@64.214.83.25' replicas will be lost NOTE: Applying these changes will result in 1 cluster transition ####################################################################### After cluster transition 1/1 ####################################################################### ============================= Membership ============================== Status Ring Pending Node ----------------------------------------------------------------------- valid 20.3% -- 'riak@192.168.17.11' valid 20.3% -- 'riak@64.214.83.26' valid 20.3% -- 'riak@64.214.83.27' valid 20.3% -- 'riak@64.214.83.28' valid 18.8% -- 'riak@64.214.83.29' ----------------------------------------------------------------------- Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 Partitions reassigned from cluster changes: 13 13 reassigned from 'riak@64.214.83.25' to 'riak@192.168.17.11' node1>
-
Commit the changes to the cluster.
riak-admin cluster commit
node1> riak-admin cluster commit Attempting to restart script through sudo -H -u riak Cluster changes committed node1>
-
Check that everything connected and functioning as expected
riak-admin member-status
node1> riak-admin member-status Attempting to restart script through sudo -H -u riak ============================= Membership ============================== Status Ring Pending Node ----------------------------------------------------------------------- valid 20.3% -- 'riak@192.168.17.11' valid 20.3% -- 'riak@64.214.83.26' valid 20.3% -- 'riak@64.214.83.27' valid 20.3% -- 'riak@64.214.83.28' valid 18.8% -- 'riak@64.214.83.29' ----------------------------------------------------------------------- Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
-
Monitor
riak-admin transfers
to make sure that all hinted handoff is complete. -
Delete the renamed ring folder to clean up once everything is done.
Repeat for the remaining nodes. Use riak@192.168.17.11
for further cluster join commands in Replace:4
riak-admin cluster join riak@192.168.17.11
node2> riak-admin cluster join riak@192.168.17.11
Attempting to restart script through sudo -H -u riak
Success: staged join request for 'riak@192.168.17.12' to 'riak@192.168.17.11'
node2>