My Docker Swarm Architecture

docker-swarm-architecture.md

This (and related gists) captures how i created my docker swarm architecture. This is intended mostly for my own notes incase i need to re-creeate anything later! As such expect some typos and possibly even an error...

Installation Step-by-Step

Each major task has its own gist, this is to help with maitainability long term.

1. Install Debian VM for each docker host
2. install Docker
3. Configure Docker Swarm
4. Install Portainer
5. Install KeepaliveD
6. glusterFS disk prep, install & config
7. gluster FS plugin for docker (optional )
8. example stack templates:
   • adguard 2 node + adguard settings sync
   • cloudflare Dynamic DNS Updater
   • infinitude carrier infinity thermostat control
   • Mosquitto MQTT
   • Nginx Proxy Manager (NPM)
   • ouath2-proxy manager
   • migrate portainer agent to be managed by portainer not recommeded
   • shepherd to update swarm images
   • traefik
   • uPoller (unifi poller)
   • watchtower
   • wordpress - todo
   • portception (portainer deployed by portainer - do not attempt)
   • auto lable nodes with name of running containers

More Details on What and Why

design goals:

• ensure every container stays running if any of the following fail (one VM, one hypervisor, one docker service)
• remove chance of blackhole requests (aka eliminate the use of DNS round robin to address the service)
• enable the use of replicated state so any container can start on any single docker swarm node and fail between nodes and see the data it needs to
• enable safe replicated shared volume across all nodes that allow state to be replicated and accessible from all nodes and allows for use of datatbases like mariadb which will corrupt if placed on NFS or CIFS/SMB shares across the network
• make it easy to backup with my synology (this model enabled me to easily backup using active backup for business)

current state 8/26/2023

• all seems to be functioning nearly a year layte
• I switched fully from native nginx container to NPM
• i elimnated NFS and iSCSI and moved all containers with state to running on GlusterFS inlcuding things with databases like wordpress
• i plan to move the VMs from Hyper-V to my new proxmox cluster

Architecture

Design Assumptions

• I wanted to continue to use docker, docker-compose, docker swarm & portainer due to existing skills
• I have no interest at this time in k8s (i don't use it at work and never will)
• Start simple, even if that means i do what i shouldn't (this is just a home network)
• This is small, the containers include (nginx reverse proxy, oauth2-proxy, wordpress site + database, mqtt, upoller, cloudflare ddns) so bear that in mind, this isn't designed for super throuput or scale - its designed for some resilliency.
• I want to deploy all services (containers) with stack templates and possibly contribute back to portainer template repo
• The clustered file system must support databases on it (like mariadb)

Design Decisions

• Debian for my docker host VMs - i seem to gel with debian and it (and other debian derivatives) seems to play nice with most contaniners
• I will only use package versions included in the debian distri (bullseye stable)
• I chosee glusterfs as my clustered, replicated file system
• Gluster volumes will be deployed in dispersed mode
• I mapped seperate VHDs into the docker hosts one for OS and one for gluster - this is to prevent risk of infinite boot loops
• my gluster service will be installed on the docker host VMs. Best practice dicates they should be seperate VMs for scale. But as all VMs share the same host CPU this really gives no benefit. If this turns out to be bad decision i will change.
• I wont tear down my current NFS and iSCSI mapped volumes (not shown) until glusterfs has been shown to run ok and survive reboots etc

A note on docker swarm and state (assume you know docker already)

Docker containers are ephemeral and generally loose all their data when they are stopped. For most docker containers there is some level of confguration state you need to pass to the container (variable, file, folders of data). Simillarly many containers want to persist data state (databases, files etc)

On a single node docker most people map a directory or file on the host into the container as a volumen or bind mount. We also see the following more advanced techniques used:

1. mount a shared CIFS or NFS volume at bootime on the docker hosts
2. defining a CIFS volume and mapping it into the container at runtime (this avoids editing fstab on the host)
3. same as aove but with NFS
4. using configs - if you have just a single, readi only, confg file that needs to be read this can be defined.

In a swarm where you want a container to run on any node you need to find a way to make the data available on all nodes in a safe effective way.

If you have a simple container that only needs environment variables to be cofigure you can do that directly when you deploy the portainer template as a portaineer stack. See this cloudflare dynamic dns updater as an example.

• Only #4 offers a safe way to make this happen (the 'config' is available to all nodes) - but this is super restrictive and doesn't help with containers that need to store more state and read/write that state. See this mosquitto mqtt example
• #1 this can work and you can mount the shares to multiple nodes via fstab. Typically databases cannot be placed on these shares and will ultimately corrupt. You do have to be careful to only have one container writing to any given file to avoid potentials issues.
• #2 and #3 - thishas the advantage of not being generall mounted to the host OS, but mount on demand by the container, this reduced all the tedious mucking about is hyperspace fstab. You do need to use the volumes UI in portaine for this.

and for nost folks NFS/CIFS shares are not replicated for high availability.

This is why in this architecture i have chose to see if I can overcome these limitations uings glusterfs.

How do you handle read/write databses with glusterfs?

How do you handle read/write databses with glusterfs?
This is my strategy

• I used disbursed not replicated volumes.
• I only ever allow a service with a database to start on one node.
• I only ever allow a service that accesses a database to start on one node
• I only ever had one service access a database (this is fine in a home lab - if i wanted more services to access the same db i would implement a true HA replicated db service where i used the native ha / replication the db offers)

in terms of implementation - any service where i need this constraint gets this in its service definition

    deploy:
      mode: replicated
      replicas: 1    

I went gluster because every database person says storing a database on NFS or CIFS and accessing will result in corruption eventually. so i didn't want to store the database on my synology and i wanted to also protect the databases from a loss of one node.

I only have see an issue once (touch wood) and that was with the nginx proxy manager mariadb - this was when i hard killed a node by accident as such this wasn't a gluster issue (I didn't need to fix gluster) and it was fixed running normal mariadb fix commands.

I have no complaints with gluster - its certainly much simpler to setup than ceph and less fragile when messing with networks - as the docker swarm is currently being moved from hyper-v to proxmox i will revist the choice of file system... if you are interested you can watch my progress in realtime here as i note my progress.... https://gist.github.com/scyto/042c8c41b23bd5ddb31d1e4e38156dff

Your Gists are very helpful to me, especially as a reference to validate my own assumptions.

Glad they are of use to folks, the more we all share and see different ways of doing things the more we all learn IMO.

I only assert 'i did it this way' never 'my way is the right way' - but i do like to learn the root pros/cons of the way i did it to figure out if there are better ways OR if the different ways are a matter of preference (too many arguments on the interwebs are poeople arguing about preferences - which is insane to me, thats like arguing is strawberry or chocolate ice cream the best - missing the point that all ice cream is awesome [for the record its strawberry of course]).

And really i write these like this and make them public - because it forces to me make half-decent record of what i did. So for example i recently came back to the gluster when i started to think about how do i expand it to a 4th node and i realized because it is a disbursed volume i can't - but because i wrote it up it was easy to figure out how to create new 4 node dispersed volume and migrate ...

I wish i had written up my steps for running Windows Hello for Business at home (no one else in the world should do this) it was one of the hardest things i ever setup in 30 years of doing systems stuff.

And really i write these like this and make them public - because it forces to me make half-decent record of what i did.

Yup, making the notes public forces one to think. Often it’s helpful for others too, comparing notes and discovering different ways to do the same thing.

I find ‘how to do stuff’ often poorly documented or ‘all over the place’; I’m writing it down to help myself, and other avoiding to waste precious time.

We should not waste our time on something trivial, that someone else could have properly documented while they were at it!

Instead weshould use our (collective) time to solve unsolved problems, so that we get even better software and platform to play with, or to go out and eat ice-cream! 😋

Do you have a write-up / notes on that? Or perhaps some useful links.

no but this guy call Drallas did and does

:-p

Do you have a write-up / notes on that? Or perhaps some useful links.

no but this guy call Drallas did and does

:-p

😊👊🏻