why-i'm-leaving-kubernetes-for-swarm.md

I have been an aggressive Kubernetes evangelist over the last few years. It has been the hammer with which I have approached almost all my deployments, and the one tool I have mentioned (shoved down clients throats) in almost all my foremost communications with clients, and it was my go to choice when I was mocking my first startup (saharacluster.com).

A few weeks ago Docker 1.13 was released and I was tasked with replicating a client's Kubernetes deployment on Swarm, more specifically testing running compose on Swarm.

And it was a dream!

All our apps were already dockerised and all I had to do was make a few modificatons to an existing compose file that I had used for testing before prior said deployment on Kubernetes.

And, with the ease with which I was able to expose our endpoints, manage volumes, handle networking, deploy and tear down the setup. I in all honesty see no reason to not use Swarm. No mission-critical feature, or incredibly convenient really nice to have feature in Kubernetes that I'm going to miss; except perhaps the Kube admin dashboard, and heapster but even those have ready replacements. Weave scope for Kube admin (admittedly not as pretty) and I could easily setup my own ELK to monitor my containers.

The moment it dawned on me how simple swarm was, was when I realised that all I had to do to expose an nginx service publicly was to publish the ports in my compose file. It hit me again when I attempted to create a number of replicas for the nginx service fully expecting to run into this Kubernetes like error Pod Deploy - Failed to fit in any node - PodFitsHostPorts that has frustrated me before, but no. It worked! It just worked! And to boot Docker intelligently loadbalanced the requests from all my nodes (nginx was accessible from all the node ips in the Swarm at port 80/443) to the various containers in the service.

Anyone who has used Kubernetes on any long-term large-scale project knows what a pain this is. If you're not on AWS or GCE and can't create a Loadbalancer service, where Kubernetes will provision an elastic ip address for your service (which you have to pay for), and you're not okay with having to access your service on a weird random port between default:30000-32767, then you have to deal with the fickle beast that is Kubernetes Ingresses. To elaborate on how many steps it takes to come close to replicating what I had achieved on Swarm with three lines in my compose file.

You'd have to do the following on Kubernetes

1. Create the ingress controller

2. Work on your Kubernetes yaml spec and define an ingress resource jumping around from the various documentation sources online

3. A bit of trial and error here to get to where you can create your ingress without error

4. Realising that to use paths i.e. example.com/path you have to create a "path" directory in the friggin /usr/share/nginx/html in the nginx container

Full docs here Kubernetes Ingresses

In short, exposing services to the outside world in Kubernetes is a pain! While with Docker however all it takes is

services:
  nginx:
    ports:
      - "80:80"
      - "443:443"
    ...

And it works! It just works!

How to use volumes in Swarm

version: '3'
volumes:
  poc:
services:
  redis:
    volumes:
      - poc:/redis

How to use volumes in Kubernetes :(

1. Create the pv sample

2. Create the pvc sample

3. Create the deployment specifying your pvc sadness sample

Configs

I am incredibly appreciative of how easily I can eyeball my entire deployment in Swarm, ports, volumes, services, dependencies, images etc etc as all the config is in one docker-compose.yml file of reasonable length as opposed to the countless files covering everything from pvs, pvcs, deployments, statefulsets etc in Kubernetes. You can define everything in one file in Kubernetes but it won't do much for readability.

[Think how many lines you need to pore through to get to the container image you're using]

Deploying and cleaning up

You have to run kubectl create -f more than once. You know it. I know it. (Unless of course you put everything into one file and trade your readability for convenience).

With compose on Swarm however, all you have to do is: docker stack deploy --compose-file=docker-compose.yml <stack-name> You could delete your app's entire namespace in Kubernetes to cleanup, but what if that's not you want? Or you didn't have the foresight to deploy your app in a separate namespace? You'd again have to run a number of kubectl delete -f commands to delete everything. You could run kubectl delete -f on a single directory with all your app's kube config files in there. You could do that. Or you could use Swarm and be able to, docker stack rm <stack-name> And have a life.

BONUS:

Pure Docker goodness

It is beyond satisying to use Docker and only Docker. To spin up a fresh vm and install Docker and only Docker. And be able to do everything you need. I have spent countless hours writing salt files and ansible playbooks to automate installing Kubernetes. You could use kargo, or kops, but all I have to do to start a Swarm cluster is install Docker and run docker Swarm init. What more could anyone want!

What Kubernetes could do:

Humans should not have to write/read config files. If there was a way I could easily deploy a Kubernetes cluster (Hint: Make up-to-date repositories for your software available from the distros repos), and not have to write any configs, that would be great.

I'm a Swarm fan (really just a fan of whatever works for you). Tips for an easier time with Swarm in 2018:

• Use 17.12 or latest stable (and try to keep it updated to latest stable). Since Swarm is key to Docker's Enterprise Edition, they are keen to fix issues, but only if you keep up with current versions. They don't backport major fixes to Edge monthly versions beyond that month.
• Many of the issues my clients and students had in 2017 were overlay networking related and resolved by 17.12 (a bit in 17.06, then 17.09, etc.). Rolling updates are now possible without packet loss.
• It's best to not run app workloads on managers in production. Keeping managers free to orchestrate and schedule the Swarm is a good thing. Make them smaller dedicated instances and use placement constraints to keep app workloads off of them. This helps prevent a runaway batch job or backup on your apps from consuming all cpu, memory, or networking accidentally on a manager and causing a manager outage.
• Always use odd number of managers to keep Raft HA.
• Don't use AWS T2 instances which are capped on CPU cycles. Raft consensus requires traffic every second to ensure health and consensus. T2's can sometimes get in the way of that with their slow/poor networking and limited CPU busts. This would show up as various issues with raft leader election and manager connection. Since switching systems to something like M3 those problems don't seem to reappear. I also wouldn't recommend anything with less then Gb networking ("High" in AWS terms).
• If you don't want Overlay networking, you can still use bridge or macvlan drivers in Swarm that have others have mentioned... it's just not the default choice so many don't realize the flexability.
• Great checklist of Docker production enterprise concerns (good for both Docker CE and Docker EE).

It was a dream when Google opened Kubernetes, but since the Docker guys started Swarm it was clear that they were creating something with usability, simplicity and power in mind. Recently I used OpenShift in a project, but realized that Swarm delivers almost all its features with much less configuration and management effort.

I've been collecting tools for managing a Swarm Cluster on the following repository:
https://github.com/flaviostutz/docker-swarm-cluster

Feel free to contribute your experience there!

Kubernetes supports direct port expose as well as direct volume allocation, exactly as docker-compose / swarm does. The reason for having services, ingresses, pvs and pvcs is managebility.
E.g. if your pod binds to host port 80, you cannot do a rolling update without downtime. You 1st need to release the port befor you bind a new instance to the port. That's why you have services - the service points always to a healty instance of your pod. The ingress exposes the service. If you don't need it, you can simply expose the port and use a different upgrade strategy.

Personally I am a swarm guy but if I had to comment on your last statement here.

What Kubernetes could do:
Humans should not have to write/read config files. If there was a way I could easily deploy a Kubernetes cluster (Hint: Make up-to-date repositories for your software available from the distros repos), and not have to write any configs, that would be great.

Terraform + EKS for AWS or Managed K8s from Azure. You can set up a production grade (self-updating) cluster with less headaches than deploying your own instances and installing K8S yourself. For local development, you can use minikube for a scaled down version of your production setup.

However, if you like Swarm you can look at my Terraform module https://registry.terraform.io/modules/trajano/swarm-aws/docker which sets up Docker Swarm on AWS, but it is self managed (hence cheaper) than Managed K8S servers theoretically.