Skip to content

Instantly share code, notes, and snippets.

@kalharbi
Last active February 22, 2022 02:58
Show Gist options
  • Save kalharbi/fd29661b9926eb087c45 to your computer and use it in GitHub Desktop.
Save kalharbi/fd29661b9926eb087c45 to your computer and use it in GitHub Desktop.
Setting up an external Zookeeper Solr Cluster

Setting up an external Zookeeper Solr Cluster

This is a step by step instruction on how to create a cluster that has three Solr nodes running in cloud mode. These instructions should work on both a local cluster (for testing) and a remote cluster where each server runs in its own physical machine. This was tested on Solr version 5.4.1 and Zookeeper version 3.4.6

Installing Solr and Zookeeper

  • Download and extract Solr:
    • curl -O http://archive.apache.org/dist/lucene/solr/5.5.3/solr-5.5.3.tgz
    • mkdir /opt/solr
    • tar -zxvf solr-5.4.1.tgz -C /opt/solr --strip-components=1
  • Download and extract ZooKeeper:
    • curl -O http://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
    • mkdir /opt/zookeeper
    • tar -zxvf zookeeper-3.4.6.tar.gz -C /opt/zookeeper --strip-components=1

Configuring a ZooKeeper quorum

  • cd /opt/zookeeper

  • Create our data directories:

    • mkdir -p /data/zookeeper/z1/data
    • mkdir -p /data/zookeeper/z2/data
    • mkdir -p /data/zookeeper/z3/data
  • Add our server ids at each data directory:

    echo 1 > /data/zookeeper/z1/data/myid
    echo 2 > /data/zookeeper/z2/data/myid
    echo 3 > /data/zookeeper/z3/data/myid
  • Copy the sample config file into the first data directory:

cp ./conf/zoo_sample.cfg /data/zookeeper/z1/zoo.cfg

  • Open /data/zookeeper/z1/zoo.cfg in your text editor and update/add the following values:

    dataDir=/data/zookeeper/z1/data
    clientPort=2181
    # Our zookeeper quorum:
    # Note: If you want to run each server in its own machine,
    # change the ip address in each entry; Obviously, you
    # do not have to use a different port number for each server.
    # each entry is defined as follows:
    # server.<value_of_myid_file_in_data_dir>:
    server.1=127.0.0.1:2222:2223
    server.2=127.0.0.1:3333:3334
    server.3=127.0.0.1:4444:4445

We configured the dataDir, which tells Zookeeper where to store its data, and clientPort, which assigns the port on which a client (Solr in this case) should use to connect to Zookeeper. Lastly, we defined the entries of our zookeeper quorum. Each entry is defined as server.id=host:port:port, where id is the server id number that we previously defined in the myid file and the host is for the IP address of the Zookeeper instance to communicate with other Zookeeper instances. The port has two parts. The first is the port that followers use to connect to the leader, and the second is for leader election. For more information, refer to Zookeeper's configuration parameters documentation.

  • Copy the config file into the data directories of the other two servers:

    cp /data/zookeeper/z1/zoo.cfg /data/zookeeper/z2/
    cp /data/zookeeper/z1/zoo.cfg /data/zookeeper/z3/
  • [This step is ONLY required when running the servers on the same host] Change both the dataDir and clientPort for the last two servers:

    • Server 2:
      $ vi /data/zookeeper/z2/zoo.cfg
      dataDir=/data/code/zookeeper/z2/data
      clientPort=2182
    • Server 3:
      $ vi /data/zookeeper/z3/zoo.cfg
      dataDir=/data/code/zookeeper/z3/data
      clientPort=2183
  • Start the servers:

     $ ./bin/zkServer.sh start /data/zookeeper/z1/zoo.cfg
     $ ./bin/zkServer.sh start /data/zookeeper/z2/zoo.cfg
     $ ./bin/zkServer.sh start /data/zookeeper/z3/zoo.cfg
  • Ensure that there are no major errors in the log file:

    • $ cat zookeeper.out

Configuring Solr

  • cd /opt/solr
  • Start three Solr instances and have them point at our Zookeeper instances:
# If you are running the Zookeeper servers on remote machines, use 
# the IP address of each server instead of the localhost.
$ ./bin/solr start -c -p 8983 -z 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
$ ./bin/solr start -c -p 8984 -z 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
$ ./bin/solr start -c -p 8985 -z 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
  • Upload our collection configuration to ZooKeeper:
$ ./server/scripts/cloud-scripts/zkcli.sh -cmd upconfig \
-zkhost 127.0.0.1:2181 \
-confdir ./server/solr/configsets/data_driven_schema_configs/conf/ \
-confname my-config
  • Create a Solr collection using the uploaded configuration.

    curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=my-colection&numShards=2&replicationFactor=1&collection.configName=my-config'

Notes:

  • If you want to create multiple collections with different schemas, then repeat the last two steps for each collection that uses a different schema. Otherwise, Zookeeper will sync the schema for all collections and you will end up with a single schema for all collections.
  • In Solr, the default maxShardsPerNode is one shard per node. In this setup, we had 3 nodes, so we should not attempt to add more replicas to a collection (e.g., numShards=2 & replicationFactor=2 will result in four shards in total spreaded across three nodes). This would cause a series of errors and crashes since two replicas of the same shard will never be allowed to exist on the same node as per the maxShardsPerNode config setting.
@almasudw3
Copy link

almasudw3 commented Oct 26, 2016

Thanks for the GIST Kalharbi, it works like a charm. :) . I have couple of question.

I have uploaded two different configurations to ZooKeeper and create two collections using below commands

My Directory structure
/opt/solr/server/solr/configsets
/tr_config
/rp_config

Upload config To ZooKeeper:

./server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -zkhost 127.0.0.1:2181 -confdir ./server/solr/configsets/tr_config/conf/ -confname tr_config

./server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -zkhost 127.0.0.1:2181 -confdir ./server/solr/configsets/rp_config/conf/ -confname rp_config

Create Collections:

curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=transactions&numShards=2&replicationFactor=1&collection.configName=tr_config'

curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=properties&numShards=2&replicationFactor=1&collection.configName=rp_config'

It works properly but then I have changed the solrconfig.xml on properties collection and update configuration again to ZooKeeper server,then it showing lockType.native on solr initialization, so then I have set lockType:none on solrconfig.xml. and it works again. but when I am pointing to another solr instances on different ports like 8983, 8984 & 8985, all are down and showing

no servers hosting shard: shard1

but If I stop the other solr instances on 8983, 8984 and 8985 port and start 8983 then all my docs are queryable. again if I start any of the 8984 or 8985 solr instance, 8983 instance again non queryable.

What is the problem ? why not able to run three instances while I am updating the configuration on ZooKeeper.?

How can I change the maxShardsPerNode ?

Thanks in Advance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment