Since many deployments may start out with 3 nodes and so little is known about how to grow a cluster from 3 memebrs to 5 members without losing the existing Quorum, here is an example of how this might be achieved.
In this example, all 5 nodes will be running on the same Vagrant host for the purpose of illustration, running on distinct configurations (ports and data directories) without the actual load of clients.
YMMV. Caveat usufructuarius.
sudo apt-get update
sudo apt-get install -y openjdk-6-jre-headless vim
mkdir ~/zook
cd ~/zook
wget http://apache.claz.org/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz # You may wish to choose a closer mirror
tar xzf zookeeper-3.4.5.tar.gz
for i in `seq 5` ; do mkdir conf$i ; echo $i > conf$i/myid ; done
tickTime=2000
dataDir=/home/vagrant/zook/conf1
clientPort=2181
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf1/zoo.cfg
tickTime=2000
dataDir=/home/vagrant/zook/conf2
clientPort=2182
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf2/zoo.cfg
tickTime=2000
dataDir=/home/vagrant/zook/conf3
clientPort=2183
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf3/zoo.cfg
Start the service, see the nodes join the cluster, snapshot the data and become active.
tickTime=2000
dataDir=/home/vagrant/zook/conf4
clientPort=2184
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
server.4=localhost:2884:3884
server.5=localhost:2885:3885
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf4/zoo.cfg
tickTime=2000
dataDir=/home/vagrant/zook/conf5
clientPort=2185
initLimit=50
syncLimit=200
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
server.4=localhost:2884:3884
server.5=localhost:2885:3885
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf5/zoo.cfg
for i in `seq 3` ;do vim conf$i/zoo.cfg ; done
server.4=localhost:2884:3884
server.5=localhost:2885:3885
Save files.
# Stop this instance with Ctrl+C, then run
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf2/zoo.cfg
Ensure that is joins the ensemble, repeat with other Follower.
# Ctrl+C
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf3/zoo.cfg
Ensure that all 4 nodes have network conenctivity to each other on the designated ports, and then bounce the Leader.
# Ctrl+C
zookeeper-3.4.5/bin/zkServer.sh start-foreground conf1/zoo.cfg
It's a good tutorial.
What is the external ip address to a zookeeper ensemble? Using a load balancer is not the right approach because load balancer doesn't know which one is the leader or which ones are up/healthy nodes.
What is your approach?