MongoDB sharding + Docker Swarm mode

mongo-docker-swarm-mode.txt

Warnings:
* as 2017-02-24 (Docker version 1.13.1), swarm mode is not officially recommeded from production. 
* There are a few bugs like swarm stop working when changing network. https://github.com/docker/docker/issues/29580
* Some kernel options (like ulimit) are still not supported on swarm when creating services https://github.com/docker/docker/issues/25209
* As 2018-09-24 docker swarm is very unstable and definitely not ready for production. Witch each new version of docker, swarm is more unstable than the previous one. I would strongly recommend avoid using swarm all together. (See issues https://github.com/moby/moby/issues/36696 and https://github.com/moby/moby/issues/37725 )
* 2021-09-10: this is an old gist with experimental configuration. Don't use this for production or anything relevant. Trust me. Also mongo versions are quite old. If you are just playing with mongo and docker swarm, go ahead.

Servers are server1 (mongos and configsrv), server2 (shard1), server3(shard2). 

-----Step 1. Create swarm-----
# ssh crimsonglory@server1 (192.168.0.5)
# docker swarm init --advertise-addr 192.168.0.5 --listen-addr 192.168.0.5
Swarm initialized: current node (e4wiamxehzk4ramu1l9nntf9d) is now a manager.

To add a worker to this swarm, run the following command:
docker swarm join \
    --token SWMTKN-1-9438ur98ewur983ju98f3j84ur989ero45oi54oi6iosrissq-airmciwkeo458u4w0486kueie8 \
    192.168.0.5:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and4follow the instructions.

-----Step 2. Add nodes to the swarm-----
Execute the command from the output of Step 1 in each node.

Now, from server1 (swarm leader) we can check nodes ids with the following command:
´´´
# docker node ls
ID                           HOSTNAME            STATUS  AVAILABILITY  MANAGER STATUS
3ebujl8qci0pk2oiuww8p5us4    server3             Ready   Active        
7dt8zo3zrhj0m9tyvwq5yes5r    server2             Ready   Active        
e4wiamxehzk4ramu1l9nntf9d *  server1             Ready   Active        Leader
´´´

-----Step 3. Create docker network-----
Create the network from the manager node.
# docker network create --driver overlay --opt encrypted mongoshard

-----Step 4. Create mongo config files-----
<repeat for server1, server2, and server3>
ssh to server.
# mkdir -p /opt/mongoshard/db-data
# chmod 777 /opt/mongoshard/db-data
put mongodb.conf in /opt/mongoshard/
The mongodb.conf is the default (same config) except:
* we need to comment the line "bindIp: 127.0.0.1".
* append the following two lines:
´´´
sharding:
   clusterRole: configsvr
´´´

put mongo-sh.conf in /opt/mongoshard/
 The mongo-sh.conf is the default (same config) except:
* we need to comment the line "bindIp: 127.0.0.1".
* append the following two lines:
´´´
sharding:
   clusterRole: shardsvr
´´´
</repeat>

-----Step 5. Create mongo shard config service -----
We want one mongocfg on the main node (The main node will be the one that runs mongos). 
# docker service create --constraint "node.id==e4wiamxehzk4ramu1l9nntf9d" --mount type=bind,src=/opt/mongoshard/db-data/,dst=/var/lib/mongodb/ --mount type=bind,src=/opt/mongoshard/mongodb.conf,dst=/etc/mongodb.conf --network mongoshard --name mongocfg1  --restart-condition on-failure --restart-max-attempts 3 --endpoint-mode dnsrr mongo:3.2 --config /etc/mongodb.conf
9kpdd7bj445d3rh0g83ajm5j4

-----Step 6. Create mongos service -----
Now we need to create mongos. mongos is a service to redirect the queries to the different mongo shards.
It ships in the same docker package as mongodb, but it is a seperate executable.
We create a file called mongos with the following content:
´´´
FROM mongo:3.2
ENTRYPOINT ["mongos"]
´´´

Then we create an image:
´´´
docker build -f mongos -t crimsonglory/mongos32 .
´´´
And we push it to our docker hub account:
´´´
docker login
´´´
Browse to hub.docker.com, login and create a repository.
´´´
docker push crimsonglory/mongos32
´´´
We only have to do this once. or zero, you can use my crimsonglory/mongos32 hub.

# docker service create --constraint "node.id==e4wiamxehzk4ramu1l9nntf9d" --network mongoshard --name mongos1 --restart-condition on-failure --restart-max-attempts 3 -p 27017:27017 crimsonglory/mongos32 --configdb mongocfg1:27017
We use only 1 config server because Docker Swarm increases latency and using more config servers will only decrease performance.
(this last command may fail if you already have Mongo installed on the main node host. That's because the container can't bind to 27017 because its on use. You may want to change the 27017:27017 to 27018:27017 or something.)

-----Step 7. Create mongo shards services -----
Now the shards:
On each host that will be used for shards do:
´´´
mkdir /opt/mongoshard/db-shard/; chmod 777 /opt/mongoshard/db-shard
´´´

´´´
# docker service create --constraint "node.id==7dt8zo3zrhj0m9tyvwq5yes5r" --network mongoshard --name mongosh1 --restart-condition on-failure --restart-max-attempts 3 --mount type=bind,src=/opt/mongoshard/db-shard,dst=/var/lib/mongodb --mount type=bind,src=/opt/mongoshard/mongo-sh.conf,dst=/etc/mongod.conf --endpoint-mode dnsrr mongo:3.2 --config /etc/mongod.conf
1l6yx68fvvjy7x7713tlwgsy3

# docker service create --constraint "node.id==3ebujl8qci0pk2oiuww8p5us4" --network mongoshard --name mongosh2 --restart-condition on-failure --restart-max-attempts 3 --mount type=bind,src=/opt/mongoshard/db-shard,dst=/var/lib/mongodb --mount type=bind,src=/opt/mongoshard/mongo-sh.conf,dst=/etc/mongod.conf --endpoint-mode dnsrr mongo:3.2 --config /etc/mongod.conf
d0xxgmihl4u2ff23awbu90brt
´´´

-----Step 8. add the mongo shards to the mongo set -----
Now tell mongos about the shards. We should be able to connect to mongos from outside the Docker container,because we map the port. We can connect from outside only if we have a mongo client installed somewhere. If not, just enter inside the container and execute mongo. (To 'enter' a container, first find out the name of the container. "docker ps | grep mongos1" to get the container name, and "docker exec -ti <containername> bash", then "mongo". This will have to be done from the server that is running the container)
´´´
mongo 127.0.0.1:27017/admin                                                                                                     
MongoDB shell version: 3.2.8
connecting to: 127.0.0.1:27017/admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2016-09-06T03:16:02.582+0000 I CONTROL  [main] ** WARNING: You are running this process as the root user, which is not recommended.
2016-09-06T03:16:02.582+0000 I CONTROL  [main] 
mongos> sh.addShard("mongosh1")
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> sh.addShard("mongosh2")
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> 
db.runCommand({"enablesharding": "DB_files"})

mongos> db.runCommand({"shardCollection": "DB_files.fs.chunks", "key": {"files_id": 1, "n": 1}, "numInitialChunks": 1000})
{ "collectionsharded" : "DB_files.fs.chunks", "ok" : 1 }
´´´

Not everything is perfect. With docker swarm mode (as 2016-10-05) we can't pass ulimit to the service. It's a pendding issue on docker repo.
Warning: do not use mongo:latest, always use versions. Because if one service goes down, when docker swarm recreates it, it will re-create the container, and it will auto-update Mongo version. So you can end up with one shard with 3.2 and other with 3.4 and it will not work.

I want to try this out 👍

why swarm mode not recommended for production @CrimsonGlory ?

why swarm mode not recommended for production @CrimsonGlory ?

@yuswitayudi Back when I tried this, swarm used to crash when the machine was under heavy load and the nodes lost connectivity (see linked github issues). About this setup, a mongo cluster inside the docker swarm provides no benefits at all, except that the containers can reach the db from inside the docker network, without exposing the db ports even to localhost.

Other downside of this setup (which I discovered this year), is that given that a shard may restart (because of failure, crash, whatever) it starts with a new container id (it is not a container restart but a new container being created at the same node). This is fine in most apps, but for a mongo cluster it pollutes the config collection, letting mongo think there are many different shards. Although this didn't cause any visible problem, it is doing now when trying to update.

why swarm mode not recommended for production @CrimsonGlory ?

@yuswitayudi Back when I tried this, swarm used to crash when the machine was under heavy load and the nodes lost connectivity (see linked github issues). About this setup, a mongo cluster inside the docker swarm provides no benefits at all, except that the containers can reach the db from inside the docker network, without exposing the db ports even to localhost.

Other downside of this setup (which I discovered this year), is that given that a shard may restart (because of failure, crash, whatever) it starts with a new container id (it is not a container restart but a new container being created at the same node). This is fine in most apps, but for a mongo cluster it pollutes the config collection, letting mongo think there are many different shards. Although this didn't cause any visible problem, it is doing now when trying to update.

I am got this error when add shard server from mongos, can U help me?

{
	"ok" : 0,
	"errmsg" : "failed to run command { isMaster: 1 } when attempting to add shard mongosh1:27017 :: caused by :: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit",
	"code" : 96,
	"codeName" : "OperationFailed",
	"operationTime" : Timestamp(1627475690, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1627475690, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}

Check if containers are up. Then check connectivity between hosts. Check that the docker ports are reachable. Then check connectivity between containers on the overlay network. Run bash inside mongos and then try to ping "mongosh1".
Make sure node.id in the example has been replaced with your values, and that all containers are on the same overlay network ("mongoshard" in the example)

its resolved with problem. But i will ask you, do you add shard on replicaset or not?

how to get the files mongodb.conf, mongo-sh.conf, shard1.conf, shard2f.conf. I need these 4 files, can you help me.

how to get the files mongodb.conf, mongo-sh.conf, shard1.conf, shard2f.conf. I need these 4 files, can you help me.

First of all. Avoid this setup on production. Trust me on this. Also mongo versions cited above are quite old. This setup is just to play with mongo & swarm. Also I would advise avoid using swarm all together and use kubernetes.

But just for the sake of knowledge. The config file of mongo config server (mongodb.conf);

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
net:
  port: 27019

sharding:
   clusterRole: configsvr

The shards file are the same, except they have a different role cluster role. You can even ad

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
net:
  port: 27019

sharding:
   clusterRole: shardsvr