Running a High Availability Service on CoreOS using Docker, Fleet, Flannel, Etcd, Confd & Nginx

README.md

Running a High Availability Service on CoreOS using Docker, Fleet, Flannel, Etcd, Confd & Nginx

• View the diagram of what's going on: https://drive.google.com/file/d/0B74l6vUHf9HrMzNrYUlyMm40RWs/view?usp=sharing

Tools used:

• coreos: server machine clustering via a shared cloud-config.yml
• etcd: key value store for service registration and discovery
• fleet: scheduling/failover of docker containers across coreos cluster
• flannel: Gives each docker container a unique ip that allows you to access the internal port (i.e. port 80 not 32679)
• confd: watch etcd for nodes arriving/leaving - template nginx configuration files / reload nginx on change

cloud-config.yml

#cloud-config

coreos:
  etcd2:
    # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
    # specify the initial size of your cluster with ?size=X
    discovery: https://discovery.etcd.io/36dbe83ce50fbac719f1f5acd5dd41e9
    # multi-region and multi-cloud deployments need to use $public_ipv4
    advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
    initial-advertise-peer-urls: http://$private_ipv4:2380
    # listen on both the official ports and the legacy ports
    # legacy ports can be omitted if your application doesn't depend on them
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://$private_ipv4:2380
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start
    - name: flanneld.service
      drop-ins:
        - name: 50-network-config.conf
          content: |
            [Service]
            ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'
      command: start

someapp-discovery@.service

[Unit]
Description=Announce Someapp%i
BindsTo=someapp@%i.service
After=someapp@%i.service

[Service]
ExecStart=/bin/sh -c "while true; do etcdctl set /services/someapp/upstream/someapp%i \"$(sleep 5 && docker inspect -f '{{.NetworkSettings.IPAddress}}' someapp%i):3000\" --ttl 60;sleep 45;done"
ExecStop=/usr/bin/etcdctl rm /services/someapp/upstream/someapp%i

[X-Fleet]
MachineOf=someapp@%i.service

someapp-lb.service

[Unit]
Description=someapp-lb
After=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill someapp-lb
ExecStartPre=-/usr/bin/docker rm someapp-lb
ExecStartPre=/usr/bin/docker pull willrstern/nginx-lb
ExecStart=/usr/bin/sh -c "/usr/bin/docker run -e SERVICE_NAME=someapp -e ETCD=\"$(ifconfig docker0 | awk '/\<inet\>/ { print $2}'):2379\" -P --name someapp-lb willrstern/nginx-lb"
ExecStartPost=/usr/bin/sh -c "sleep 3 && curl -X PUT 159.203.110.244:2379/v2/keys/subdomains/someapp -d value=$(/usr/bin/ip route|grep 'eth0.\+src' | head -1 | grep -o '[0-9\.]\+\s\+$' | grep -o '[0-9\.]\+'):$(/usr/bin/docker ps | grep 'someapp-lb' | grep -o '[0-9]\+->' | grep -o '[0-9]\+')"
ExecStop=/usr/bin/docker stop someapp-lb

someapp@.service

[Unit]
Description=someapp%i
After=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill someapp%i
ExecStartPre=-/usr/bin/docker rm someapp%i
ExecStartPre=/usr/bin/docker pull willrstern/node-sample
ExecStart=/usr/bin/docker run -e APPNAME=someapp%i --name someapp%i -P willrstern/node-sample
ExecStop=/usr/bin/docker stop someapp%i