Adding nodes to a ZooKeeper ensemble

zook_grow.md

Adding 2 nodes to an existing 3-node ZooKeeper ensemble without losing the Quorum

Since many deployments may start out with 3 nodes and so little is known about how to grow a cluster from 3 memebrs to 5 members without losing the existing Quorum, here is an example of how this might be achieved.

In this example, all 5 nodes will be running on the same Vagrant host for the purpose of illustration, running on distinct configurations (ports and data directories) without the actual load of clients.

YMMV. Caveat usufructuarius.

Step 1: Have a healthy 3-node ensemble

Ensure all 3 nodes are up, one is the leader, and all are in sync

setup

sudo apt-get update
sudo apt-get install -y openjdk-6-jre-headless vim

mkdir ~/zook
cd ~/zook
wget http://apache.claz.org/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz # You may wish to choose a closer mirror
tar xzf zookeeper-3.4.5.tar.gz
for i in `seq 5` ; do mkdir conf$i ; echo $i > conf$i/myid ; done

tickTime=2000
dataDir=/home/vagrant/zook/conf1
clientPort=2181
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883

zookeeper-3.4.5/bin/zkServer.sh start-foreground conf1/zoo.cfg


tickTime=2000
dataDir=/home/vagrant/zook/conf2
clientPort=2182
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883

zookeeper-3.4.5/bin/zkServer.sh start-foreground conf2/zoo.cfg


tickTime=2000
dataDir=/home/vagrant/zook/conf3
clientPort=2183
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883

zookeeper-3.4.5/bin/zkServer.sh start-foreground conf3/zoo.cfg

Step 2: Set up 2 new nodes to join the cluster

Start the service, see the nodes join the cluster, snapshot the data and become active.

tickTime=2000
dataDir=/home/vagrant/zook/conf4
clientPort=2184
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
server.4=localhost:2884:3884
server.5=localhost:2885:3885

zookeeper-3.4.5/bin/zkServer.sh start-foreground conf4/zoo.cfg


tickTime=2000
dataDir=/home/vagrant/zook/conf5
clientPort=2185
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
server.4=localhost:2884:3884
server.5=localhost:2885:3885

zookeeper-3.4.5/bin/zkServer.sh start-foreground conf5/zoo.cfg

Step 3: Add the 2 new nodes config to existing cluster

for i in `seq 3` ;do vim conf$i/zoo.cfg ; done

server.4=localhost:2884:3884
server.5=localhost:2885:3885

Save files.

Step 4: Restart Followers with new config

# Stop this instance with Ctrl+C, then run
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf2/zoo.cfg

Ensure that is joins the ensemble, repeat with other Follower.

# Ctrl+C
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf3/zoo.cfg

Step 5: Restart the Leader

Ensure that all 4 nodes have network conenctivity to each other on the designated ports, and then bounce the Leader.

# Ctrl+C
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf1/zoo.cfg

What about the reverse? We're working on scaling a ZK cluster DOWN and worrying about how to avoid losing quorum when nodes are removed. In a cluster of 6 (yes, I know we should run an odd-numbered cluster. Inherited setup), we're moving to 3. If we remove 3 nodes, quorum won't be established and everything goes sideways. So how do we establish a smaller (temporarily dangerous) quorum of 2, then remove 3 old nodes? If we update a config on one of the machines to 3 nodes, it will disagree with the rest of cluster...then what? We can't shut down/restart all the nodes at once either. Anyone have any thoughts here?

Still digging to try to find answers...

Would something like this work? I would start by updating config on clients to the smaller set. Maybe that will help the disagreement problem you mention by making sure that any potential consistency issues on the servers being removed won't matter.

I would think that the disagreement risk would be if followers think they are in the ensemble when reads come in, but the leader doesn't think those nodes are in the ensemble when writes happen. That could lead to reads from stale data. If the clients don't send requests to those nodes due to having the updated list, might that mitigate any potential problems?

After updating clients, I would update the 3 servers you are keeping with the new list and then shut down the 3 that are being deprecated.

Disclaimer: this is based on extrapolating from documentation I only read earlier today. I have done quite a few ZK cut-overs to new nodes, but I did so no background in ZooKeeper.