Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?

I have been an aggressive Kubernetes evangelist over the last few years. It has been the hammer with which I have approached almost all my deployments, and the one tool I have mentioned (shoved down clients throats) in almost all my foremost communications with clients, and it was my go to choice when I was mocking my first startup (saharacluster.com).

A few weeks ago Docker 1.13 was released and I was tasked with replicating a client's Kubernetes deployment on Swarm, more specifically testing running compose on Swarm.

And it was a dream!

All our apps were already dockerised and all I had to do was make a few modificatons to an existing compose file that I had used for testing before prior said deployment on Kubernetes.

And, with the ease with which I was able to expose our endpoints, manage volumes, handle networking, deploy and tear down the setup. I in all honesty see no reason to not use Swarm. No mission-critical feature, or incredibly convenient really nice to have feature in Kubernetes that I'm going to miss; except perhaps the Kube admin dashboard, and heapster but even those have ready replacements. Weave scope for Kube admin (admittedly not as pretty) and I could easily setup my own ELK to monitor my containers.

The moment it dawned on me how simple swarm was, was when I realised that all I had to do to expose an nginx service publicly was to publish the ports in my compose file. It hit me again when I attempted to create a number of replicas for the nginx service fully expecting to run into this Kubernetes like error Pod Deploy - Failed to fit in any node - PodFitsHostPorts that has frustrated me before, but no. It worked! It just worked! And to boot Docker intelligently loadbalanced the requests from all my nodes (nginx was accessible from all the node ips in the Swarm at port 80/443) to the various containers in the service.

Anyone who has used Kubernetes on any long-term large-scale project knows what a pain this is. If you're not on AWS or GCE and can't create a Loadbalancer service, where Kubernetes will provision an elastic ip address for your service (which you have to pay for), and you're not okay with having to access your service on a weird random port between default:30000-32767, then you have to deal with the fickle beast that is Kubernetes Ingresses. To elaborate on how many steps it takes to come close to replicating what I had achieved on Swarm with three lines in my compose file.

You'd have to do the following on Kubernetes

  1. Create the ingress controller

  2. Work on your Kubernetes yaml spec and define an ingress resource jumping around from the various documentation sources online

  3. A bit of trial and error here to get to where you can create your ingress without error

  4. Realising that to use paths i.e. example.com/path you have to create a "path" directory in the friggin /usr/share/nginx/html in the nginx container

Full docs here Kubernetes Ingresses

In short, exposing services to the outside world in Kubernetes is a pain! While with Docker however all it takes is

services:
  nginx:
    ports:
      - "80:80"
      - "443:443"
    ...

And it works! It just works!

How to use volumes in Swarm

version: '3'
volumes:
  poc:
services:
  redis:
    volumes:
      - poc:/redis

How to use volumes in Kubernetes :(

  1. Create the pv sample

  2. Create the pvc sample

  3. Create the deployment specifying your pvc sadness sample

Configs

I am incredibly appreciative of how easily I can eyeball my entire deployment in Swarm, ports, volumes, services, dependencies, images etc etc as all the config is in one docker-compose.yml file of reasonable length as opposed to the countless files covering everything from pvs, pvcs, deployments, statefulsets etc in Kubernetes. You can define everything in one file in Kubernetes but it won't do much for readability.

[Think how many lines you need to pore through to get to the container image you're using]

Deploying and cleaning up

You have to run kubectl create -f more than once. You know it. I know it. (Unless of course you put everything into one file and trade your readability for convenience).

With compose on Swarm however, all you have to do is: docker stack deploy --compose-file=docker-compose.yml <stack-name> You could delete your app's entire namespace in Kubernetes to cleanup, but what if that's not you want? Or you didn't have the foresight to deploy your app in a separate namespace? You'd again have to run a number of kubectl delete -f commands to delete everything. You could run kubectl delete -f on a single directory with all your app's kube config files in there. You could do that. Or you could use Swarm and be able to, docker stack rm <stack-name> And have a life.

BONUS:

Pure Docker goodness

It is beyond satisying to use Docker and only Docker. To spin up a fresh vm and install Docker and only Docker. And be able to do everything you need. I have spent countless hours writing salt files and ansible playbooks to automate installing Kubernetes. You could use kargo, or kops, but all I have to do to start a Swarm cluster is install Docker and run docker Swarm init. What more could anyone want!

What Kubernetes could do:

Humans should not have to write/read config files. If there was a way I could easily deploy a Kubernetes cluster (Hint: Make up-to-date repositories for your software available from the distros repos), and not have to write any configs, that would be great.

Have you looked at Rancher? It uses libcompose as the base for it's Cattle orchestration, and adds in a few additions, but the syntax is the same.

I also support k8s orchestration, and spins all the supporting services up without having to write any configs.

The UI and API is nice too.

Owner

jonathan-kosgei commented Feb 25, 2017

@ecliptik admittedly I haven't, though there's a lot of buzz around it. Sounds like an interesting solution, to be able to use docker-compose syntax on kubernetes. There's also kompose https://kubernetes.io/docs/tools/kompose/ which is pretty much what its name implies.

Kubernetes, Swarm, etc. are just tools to attain a business goal. However, whatever Docker Inc. does with Swarm, many people will stay with Kubernetes:

  1. No vendor lock-in. (i.e. support for rkt, etc. available for k8s). Only docker, docker, docker is bad for the container ecosystem. Vendor independent orchestration is the way forward.
  2. K8s has very good admin tools, which is very important to businesses.
  3. Also k8s has strong developments around serverless (e.g. fission) and orchestrating vm's. How's Docker Swarm doing here?

Swarm looks to me a quite nice tool for everyone who only lives in Dockerland. When one's world is wider, Kubernetes remains for now the way to go.

You have to run kubectl create -f more than once. You know it. I know it.

kubectl apply -R -f

puja108 commented Feb 28, 2017

Most of what you mention is on the operations side of things. If you get Kubernetes managed and Ingress Controller and Storage Classes have been set up for you, all that is left is managing the admittedly huge amount of YAMLs and all the boilerplate that comes with them. There's kompose (mentioned above) that is being worked on more recently, and that basically does what you want above. I know there's also other efforts by various people in the community to make deployment definitions easier.

Generally, I agree though that UX is bad in Kubernetes, there's a general lack of "making it easy for the user" and also lots of confusion around where to find the fitting documentation (e.g. with an NGINX Ingress Controller you have to look at 3 different places and at the right commits for your deployed release to find the right flags you can set). It seems there never was a focus on this. It's something we as a community need to work on more.

What I fear with Docker Swarm is on the one side the lock-in and with that being helplessly exposed to any decision Docker will make in the future, be that what you need or not. On the other side it's also that Kubernetes has a reason for its complexity and the power it gives you with things like being able to have all kinds of other services (e.g. headless) make it able to cope with lots of strange use cases that are out of the ordinary.

srgrn commented Mar 6, 2017

Well, I have a quick question, how did you made the volumes work for you?
I have rebuilt the development environment i use and i have the latest docker engine and compose however it keeps saying that volumes are not supported in swarmmode, and it is similar to what the docs are specifying.

parity3 commented Mar 8, 2017

The drawbacks of vendor lock-in are reduced when the vendor is providing the service to you both for free, and open-source. There is the reasoning that the vendor's agenda can influence the swarm project's direction to sway you toward using Docker Enterprise.. but as soon as swarm becomes less usable as a free service, people will be able to run their containers just fine on rancher/kube/etc without much headache? The idea is to engineer your applications/platform so that it keeps all of its abstractions at the docker container level (at least the ones that can exist there) instead of higher up; then you won't be locked in by swarm or any other container-management-management system. To sum up, I suspect that for many situations what kube/rancher/others excel at, are either not being used correctly or not needed at all for many app platforms/companies. I would guess that a lot of that differentiation comes from the apparent need for massive horizontal scaling performance (I mean, it's so cool to be a big boy right?) but that is really a think-about-later task vs other more immediately pressing needs/benefits docker can fulfill like team collaboration/versioning/deployment/security/automation/devops/redundancy/failure tolerance/dependency separation/etc...

Have you tried Portainer.io as a front end UI for Swarmmode? Makes swarm insanely easy to use..

aofry commented Mar 9, 2017

I never got compose. Its nice for dev & POC but when you get to micro services describing your full system in one file with the load balancer configuration is just a big pile.
In my point of view, Docker guys are building the tooling for Dev and Google and K8S guys are building the tooling for ops (or production mindset) first.

Docker swarm is easy to get started with. Trouble starts when you are performance testing a solution that leverages routing mesh and networking. We were running in to issues related to routing mesh. These issues appear in 15 minute load tests with 50 concurrent users. Upgrade to the latest version of Swarm Docker version 17.03 as it resolved many issues for us.

I sugest to have a look at the changelog of the swarm project and the Release page on github. There is no update since 4 month.
In contrast, kubernetes is an ongoing project. Nearly every week kubernetes has a new release

ssboisen commented Jun 1, 2017

@HusseinMorsy That's docker swarm, not docker swarm mode. Someone at docker should have thought twice about naming the two similar but different products almost identically.

bprashanth commented Jun 9, 2017

Fyi this

services:
  nginx:
    ports:
      - "80:80"
      - "443:443"
    ...

is possible on kube too, eg: https://github.com/kubernetes/kubernetes/blob/master/examples/https-nginx/nginx-app.yaml#L10

Ingress solves a slightly different problem. Just pointing out that you don't need to use it if you aren't interested in, say, a cross platform abstraction to help with the nginx.conf -> cloud CDN aspect ratio, or federating clusters once you land in the clouds.

How to use volumes in Kubernetes :(
Create the pv sample
Create the pvc sample
Create the deployment specifying your pvc sadness sample

You don't have to create either if you bring up the cluster with a volume provisioner, the provisioner will just stamp out volumes/claims for you based on the specified storage-class (eg: https://github.com/kubernetes/contrib/blob/master/pets/zookeeper/zookeeper.yaml#L114).

killcity commented Aug 4, 2017

I'm on the swarm side as of now. That doesn't mean I won't switch to k8s at some point. Here's why: I'm surprised more people haven't embraced macvlan. Fortunately in Docker 17.0.6, macvlan in swarm was finally released. No unnecessary load balancers (dns with health checks via consul is faster), no extra L3 hops, increasing latency. Everything is just routable and "there". Each container has it's own real MAC and IP. For latency sensitive stacks, this is the way to go. You can also still use your own service discovery tools (ie: consul) and not have deal with the extra bits of internal dns and built-in service discovery that only work within the unroutable overlay networks, etc. I know macvlan support is apparently available for k8s, but I haven't heard of many people leveraging it. Not to mention, there aren't 20 containers running to support this, not knowing what half of them are doing, as you'd get with k8s.

I feel like k8s is the new Openstack. Everyone is told they need to run it, but no one really knows why. Swarm took me 5 minutes to get running. Try that with k8s, especially with kubeadm, tectonic, rancher, etc.

Your last post here interests me. At my company we just dumped k8s for swarm almost entirely due to the complexity issues you've outlined. k8s is just so hard to stand up with so many parts we found ourselves using tectonic and rancher to do that lifting only to realize we didn't really understand the platform when it came time to troubleshoot and if said tool had an issue (looking at you, rancher) we were hammered.

However I'm very interested in reading more about macvlan and consul and how you are using that to supplement swarm. While the swarm mode is dirt simple to setup (docker swarm init .... done. Seriously that easy), we are exploring broader service discovery tools and well ... sounds like you may have forged that path already.

Any links you are willing to provide?

industrialsynthfreak commented Sep 8, 2017

Hi, not so experienced swarm user here.

Well, my three current problems with the swarm mode:

  • it's still in active development: features emerge, docs change, bugs appear. For example, there's still no start delay argument in the compose file healthcheck config; "depends_on" feature doesn't work in the swarm mode, and "stack deploy" won't tell you about that - you need to read the docs carefully
  • services auto-scaling ? (no easy way here)
  • occasional connection loss between master and worker nodes (still haven't figured why this happens)

So, this may be a nice technology, but one should use the swarm carefully when going into prod.

saada commented Sep 9, 2017

@industrialsynthfreak ... for autoscaling you can try Orbiter

I think you should not leave. Kubernetes has introduced kcompose - https://kubernetes.io/docs/tools/kompose/user-guide/#warning-about-deployment-configs

sasavilic commented Dec 8, 2017

I wish you the best luck! We have been running a several (dev/test/prod) Swarm clusters for 8 months and during that time our clusters failed apart multiple times. In the end, you get error like "dispatcher is down" and you don't have any idea what is going on. The only solution was to restart all nodes, one after another, hoping that things will fix it self. If it didn't work, then we had to remove all nodes from swarm, rebuild swarm and redeploy software. We also found a few bugs in swarm (i.e. moby/moby#33685) that are still not fixed.

We also used a nginx as reverse proxy + additional watch script to re-generate nginx config when something is deployed/removed from cluster. But since nginx will not start if it can't resolve dns name of upstream, our script had to make sure that name of service is resolvable before adding it to nginx. Yet, few times I have observed that a my reverse proxies on different node would have different configuration. I first thought it was error in script, but further manual inspection (jumping into container and executing dns resolve commands) showed that dns resolving on some nodes didn't work properly. Solution, restart all nodes, one after another, 2 times (restarting once didn't help).

Reporting such bugs is nightmare for us and nightmare for docker swarm maintainers: we don't know how to reproduce them. And even for bugs that are clearly reproducible like one a mentioned above, there is no fix for a months.

After we have switched to k8s, I haven't see cluster any cluster failure so far. The thing that I really like about k8s is that they follow simple UNIX philosophy: do the one thing and do it right. I have clear understanding what api server does, what is kube-proxy for, kubelet, etcd, etc. And if something is wrong, I have idea where to look for. With docker swarm, when something goes wrong you don't have any clue whatsoever what is going on. Only solution was to restart docker deamon itself, but that meant that all running instances on that node had to be restarted or even moved to other nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment