Cloud Init script for building a coreos cluster at home on XenServer using Cloud-Init and ETCD2

README.md

This Cloud-init script can be used to bootstrap a Xen based CoreOS server through Xen orchestra.

Features;

1. Workaround for the lack of /etc/environment on 'unsupported' hypervisors
2. Static IP allocation to deal with dhcp causing problems (On my network at least)
3. etcd2 bootstrap discovery using public service and discovery tokens
   1. Manual reconfiguration to allow for membership changes after cluster is operational

Usage

1. Get discovery token curl -w "\n" 'https://discovery.etcd.io/new?size=3'
2. Boot 3 CoreOS servers using a CoreOS ISO live CD, ensuring that for each you've set
   1. The Token obtained in (1)
   2. A different, and avaliable, static IP address
3. Check the ETCD2 cluster status using etcdctl cluster-health
4. If all nodes are operational, carry on - otherwise troubleshoot and get ETCD2 working
5. install coreos using `coreos-install -d /dev/xvda -o xen -C stable'
6. Remove the now broken node from the etcdcluster, eg etcdctl member remove 4ab6f50e40411ab9
7. Add a new node with the same name for the rebooted (and empty etcd node), eg etcdctl member add coreosred http://192.168.1.15:2380
8. Copy the output of step (7) into a file named /etc/default/etcd2
   1. Also append ETCD_DISCOVERY= to Unset the discovery variable set by the cloud-init script
9. Reboot the server and check cluster health - if all goes well, your properly installed coreos node should now be running etcd2 consistently

Repeat the above steps for the remaining two nodes in your cluster.

Example of creating /etc/default/etcd2

1. On a etcd2 operational node; etcdctl member remove 7063a17f2961acfa && etcdctl member add coreosblue http://192.168.1.16:2380
2. On the node you just added (here; coreosblue);

cat << EOF > /etc/default/etcd2
> ETCD_NAME="coreosblue"
> ETCD_INITIAL_CLUSTER="coreosyellow=http://192.168.1.17:2380,coreosblue=http://192.168.1.16:2380,coreosred=http://192.168.1.15:2380"
> ETCD_INITIAL_CLUSTER_STATE="existing"
> ETCD_DISCOVERY=
> EOF

cloud-init

#cloud-config

hostname: %VMNAMETOHOSTNAME%
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy8xNHI82UuCSWa72GcoSXjYWoDk2B3rA7bVPzx87/i3VodRGh4EAd2gs6w2Mts3AgFtlXrRvcPQEQyE4HOCibrHQOh1IucXh+iMs8KQFXJ8yPYs0QJUZRdrM+cOfyIyOhaiSjZCgACKW851HUzCUZt9KXWlWnNUcpnbFY43uhxJ7Q1urM/Qp8gO9clhFd7UDJCHSQXiImrIGCtuh7IENLWqrlH+Fgafb+3S6u/5CYU8+XtV9SD4fzHOn7vXSMk5nSCpFKIhscGMj1zHquZcKqv38cO8ZbmVn5LUMsDkSEXg2XET4ZH03Zl6TR3wpIexHtIzjxctLvKVG6jvwaz303 toby@Tobys-MacBook-Pro.local
  # The following entry will automatically be replaced with a public key
  # generated by container management plugin. The key-entry must exist,
  # in order to enable container management for this VM.
  - ssh-rsa %CONTAINERRSAPUB%

coreos:
  fleet:
    public-ip: $public_ipv4
  flannel:
    interface: $public_ipv4
  update:
    reboot-strategy: reboot
  units:
  # Generate a new token for discovery using $(curl -w "\n" 'https://discovery.etcd.io/new?size=3')
  - name: etcd-env-generator.service
    command: start
    content: |
      [Unit]
      Description=Creates an EnvironmentFile with etcd2 setup on private network to be injected into etcd2 service
      Documentation=https://github.com/pavlo/coreos-utils/etcd-env-generator
      Requires=network-online.target
      After=network-online.target
      [Service]
      ExecStart=/tmp/etcd-env-generator.sh eth0 1d95959f0021734961d5b7264aa46bbf
      RemainAfterExit=yes
      Type=oneshot
  - name: etcd2.service
    command: start
    drop-ins:
    - name: "10-bootstrap.conf"
      content: |
        [Unit]
        Requires=etcd-env-generator.service
        After=etcd-env-generator.service
        [Service]
        EnvironmentFile=/etc/default/etcd2-bootstrap
        EnvironmentFile=-/etc/default/etcd2
  - name: fleet.service
    command: start
  # Hypervisor Linux Guest Agent
  - name: xe-linux-distribution.service
    command: start
    content: |
      [Unit]
      Description=Hypervisor Linux Guest Agent
      After=docker.service

      [Service]
      ExecStartPre=/media/configdrive/agent/xe-linux-distribution /var/cache/xe-linux-distribution
      ExecStart=/media/configdrive/agent/xe-daemon
  - name: 00-eth0.network
    runtime: true
    content: |
      [Match]
      Name=eth*

      [Network]
      Address=192.168.1.17/24
      Gateway=192.168.1.1
      DNS=192.168.1.1

write_files:
  # Enable ARP notifications for smooth network recovery after migrations
  - path: /etc/sysctl.d/10-enable-arp-notify.conf
    permissions: 0644
    owner: root
    content: |
      net.ipv4.conf.all.arp_notify = 1
  # Dynamically configure the ETCD env so it uses priv IP's and our token
  - path: /tmp/etcd-env-generator.sh
    permissions: 0744
    owner: root
    content: |    
      #!/bin/sh

      set -e

      IFACE=${1:-eth0}
      CLUSTER_TOKEN=${2}
      TARGET=/etc/default/etcd2-bootstrap

      IP=`ifconfig $IFACE | grep -m 1 inet | awk '{print $2}'`
      URL="http://${IP}"
      
      echo "Creating ${TARGET} file with etcd2 configuration to be available on ${IP} address"
      touch ${TARGET}
      
      echo "ETCD_NAME=$(hostname)" > ${TARGET}
      echo "ETCD_DISCOVERY=https://discovery.etcd.io/${CLUSTER_TOKEN}" >> ${TARGET}
      echo "ETCD_ADVERTISE_CLIENT_URLS=${URL}:2379" >> ${TARGET}
      echo "ETCD_INITIAL_ADVERTISE_PEER_URLS=${URL}:2380" >> ${TARGET}
      echo "ETCD_LISTEN_CLIENT_URLS=${URL}:2379,${URL}:4001,http://127.0.0.1:2379,http://127.0.0.1:4001" >> ${TARGET}
      echo "ETCD_LISTEN_PEER_URLS=${URL}:2380" >> ${TARGET}

      # This deals with https://github.com/coreos/bugs/issues/65
      echo "COREOS_PUBLIC_IPV4=${IP}"   > /etc/environment
      echo "COREOS_PRIVATE_IPV4=${IP}" >> /etc/environment

sorry I was on 1395 for a lot of time now and I didn't upgrade, so I have a few questions. Do you have some time to discuss soon about your usage of corseos?