Skip to content

Instantly share code, notes, and snippets.

@learncodeacademy
Last active June 17, 2021 15:43
Show Gist options
  • Save learncodeacademy/2a11bd906c87cf2301a1 to your computer and use it in GitHub Desktop.
Save learncodeacademy/2a11bd906c87cf2301a1 to your computer and use it in GitHub Desktop.
Running a High Availability Service on CoreOS using Docker, Fleet, Flannel, Etcd, Confd & Nginx

Running a High Availability Service on CoreOS using Docker, Fleet, Flannel, Etcd, Confd & Nginx

Tools used:

  • coreos: server machine clustering via a shared cloud-config.yml
  • etcd: key value store for service registration and discovery
  • fleet: scheduling/failover of docker containers across coreos cluster
  • flannel: Gives each docker container a unique ip that allows you to access the internal port (i.e. port 80 not 32679)
  • confd: watch etcd for nodes arriving/leaving - template nginx configuration files / reload nginx on change
#cloud-config
coreos:
etcd2:
# generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
# specify the initial size of your cluster with ?size=X
discovery: https://discovery.etcd.io/36dbe83ce50fbac719f1f5acd5dd41e9
# multi-region and multi-cloud deployments need to use $public_ipv4
advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
initial-advertise-peer-urls: http://$private_ipv4:2380
# listen on both the official ports and the legacy ports
# legacy ports can be omitted if your application doesn't depend on them
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
listen-peer-urls: http://$private_ipv4:2380
units:
- name: etcd2.service
command: start
- name: fleet.service
command: start
- name: flanneld.service
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'
command: start
[Unit]
Description=Announce Someapp%i
BindsTo=someapp@%i.service
After=someapp@%i.service
[Service]
ExecStart=/bin/sh -c "while true; do etcdctl set /services/someapp/upstream/someapp%i \"$(sleep 5 && docker inspect -f '{{.NetworkSettings.IPAddress}}' someapp%i):3000\" --ttl 60;sleep 45;done"
ExecStop=/usr/bin/etcdctl rm /services/someapp/upstream/someapp%i
[X-Fleet]
MachineOf=someapp@%i.service
[Unit]
Description=someapp-lb
After=docker.service
[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill someapp-lb
ExecStartPre=-/usr/bin/docker rm someapp-lb
ExecStartPre=/usr/bin/docker pull willrstern/nginx-lb
ExecStart=/usr/bin/sh -c "/usr/bin/docker run -e SERVICE_NAME=someapp -e ETCD=\"$(ifconfig docker0 | awk '/\<inet\>/ { print $2}'):2379\" -P --name someapp-lb willrstern/nginx-lb"
ExecStartPost=/usr/bin/sh -c "sleep 3 && curl -X PUT 159.203.110.244:2379/v2/keys/subdomains/someapp -d value=$(/usr/bin/ip route|grep 'eth0.\+src' | head -1 | grep -o '[0-9\.]\+\s\+$' | grep -o '[0-9\.]\+'):$(/usr/bin/docker ps | grep 'someapp-lb' | grep -o '[0-9]\+->' | grep -o '[0-9]\+')"
ExecStop=/usr/bin/docker stop someapp-lb
[Unit]
Description=someapp%i
After=docker.service
[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill someapp%i
ExecStartPre=-/usr/bin/docker rm someapp%i
ExecStartPre=/usr/bin/docker pull willrstern/node-sample
ExecStart=/usr/bin/docker run -e APPNAME=someapp%i --name someapp%i -P willrstern/node-sample
ExecStop=/usr/bin/docker stop someapp%i
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment