Running Kubernetes Example on CoreOS, Part 2 with flannel (formerly Rudder)

README.md

Running Kubernetes Example on CoreOS, Part 2 with flannel (formerly Rudder)

• https://coreos.com/blog/running-kubernetes-example-on-CoreOS-part-2/
• https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/guestbook/README.md
• https://coreos.com/blog/introducing-rudder/
• https://github.com/coreos/flannel

Step Zero: Build up a Kubernetes cluster

$ git clone https://gist.github.com/6177b69f1754f0590dbe.git
$ vagrant up

It will boot up one for etcd discovery, one master and two minion servers (minion-x).

Setup an SSH tunnel

Setup an SSH tunnel to the Kubernetes API Server in order to use kubecfg on your local machine.

$ curl -OL http://storage.googleapis.com/kubernetes/darwin/kubecfg
$ chmod +x kubecfg
$ mv kubecfg /usr/local/bin
$ kubecfg --version
Kubernetes v0.2-dev

$ vagrant ssh-config master > ssh.config
$ ssh -f -nNT -L 8080:127.0.0.1:8080 -F ssh.config master
$ kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------

Cf.) https://github.com/kelseyhightower/kubernetes-coreos#running-commands-remotely

Step One: Turn up the redis master

$ kubecfg -c redis-master.json create pods
I0919 21:28:53.758488 29403 request.go:292] Waiting for completion of /operations/1
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    /                   name=redis-master   Waiting

$ kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    192.168.12.11/      name=redis-master   Waiting

$ kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    192.168.12.11/      name=redis-master   Running

Step Two: Turn up the master service

$ kubecfg -c redis-master-service.json create services
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000

$ kubecfg list services
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000

Step Three: Turn up the replicated slave pods

$ kubecfg -c redis-slave-controller.json create replicationControllers
I0919 21:35:28.725306 29856 request.go:292] Waiting for completion of /operations/4
ID                     Image(s)                   Selector            Replicas
----------             ----------                 ----------          ----------
redisSlaveController   brendanburns/redis-slave   name=redisslave     2

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Waiting
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Waiting

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Running
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Running

Step Four: Create the redis slave service

$ kubecfg -c redis-slave-service.json create services
I0919 21:46:52.782728 30375 request.go:292] Waiting for completion of /operations/9
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redisslave          name=redisslave     name=redisslave     10001

$ kubecfg list services
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000
redisslave          name=redisslave     name=redisslave     10001

Step Five: Create the frontend pod

$ kubecfg -c frontend-controller.json create replicationControllers
I0919 21:47:38.445604 30429 request.go:292] Waiting for completion of /operations/10
ID                   Image(s)                 Selector            Replicas
----------           ----------               ----------          ----------
frontendController   brendanburns/php-redis   name=frontend       2

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Running
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Running
3f3fc79a-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.12/      name=frontend,replicationController=frontendController       Waiting
3f3ff5ff-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.11/      name=frontend,replicationController=frontendController       Waiting
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Running
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Running
3f3fc79a-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.12/      name=frontend,replicationController=frontendController       Running
3f3ff5ff-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.11/      name=frontend,replicationController=frontendController       Running

$ open http://192.168.12.11:8000
$ open http://192.168.12.12:8000

discovery.yml

#cloud-config

coreos:
  etcd:
    addr: $public_ipv4:4001
    peer-addr: $public_ipv4:7001

  units:
    - name: etcd.service
      command: start

frontend-controller.json

{
  "id": "frontendController",
  "kind": "ReplicationController",
  "apiVersion": "v1beta1",
  "desiredState": {
    "replicas": 2,
    "replicaSelector": {"name": "frontend"},
    "podTemplate": {
      "desiredState": {
         "manifest": {
           "version": "v1beta1",
           "id": "frontendController",
           "containers": [{
             "name": "php-redis",
             "image": "brendanburns/php-redis",
             "ports": [{"containerPort": 80, "hostPort": 8000}]
           }]
         }
       },
       "labels": {"name": "frontend"}
      }},
  "labels": {"name": "frontend"}
}

master.yml

#cloud-config

coreos:
  etcd:
    discovery: http://%ETCD_DISCVERY%:4001/v2/keys/cluster
    addr: $public_ipv4:4001
    peer-addr: $public_ipv4:7001

  units:
    - name: etcd.service
      command: start

    - name: flannel.service
      command: start
      content: |
        [Unit]
        Requires=etcd.service
        After=etcd.service

        [Service]
        ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.100.0.0/16"}'
        ExecStart=/opt/bin/flanneld -iface=$public_ipv4

        [Install]
        WantedBy=multi-user.target

    - name: docker.service
      command: restart
      content: |
        [Unit]
        Description=Docker Application Container Engine
        Documentation=http://docs.docker.io
        Requires=flannel.service
        After=flannel.service

        [Service]
        EnvironmentFile=/run/flannel/subnet.env
        ExecStartPre=-/usr/bin/ip link set dev docker0 down
        ExecStartPre=-/usr/sbin/brctl delbr docker0
        ExecStart=/usr/bin/docker -d -s=btrfs -H fd:// --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
        Restart=on-failure
        RestartSec=5

        [Install]
        WantedBy=multi-user.target

    - name: download-kubernetes.service
      command: start
      content: |
        [Unit]
        After=network-online.target
        Before=apiserver.service
        Before=controller-manager.service
        Description=Download Kubernetes Binaries
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=network-online.target

        [Service]
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/apiserver
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/controller-manager
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/kubecfg
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/scheduler
        ExecStart=/usr/bin/chmod +x /opt/bin/apiserver
        ExecStart=/usr/bin/chmod +x /opt/bin/controller-manager
        ExecStart=/usr/bin/chmod +x /opt/bin/kubecfg
        ExecStart=/usr/bin/chmod +x /opt/bin/scheduler
        RemainAfterExit=yes
        Type=oneshot

    - name: apiserver.service
      command: start
      content: |
        [Unit]
        After=etcd.service
        After=download-kubernetes.service
        ConditionFileIsExecutable=/opt/bin/apiserver
        Description=Kubernetes API Server
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Wants=etcd.service
        Wants=download-kubernetes.service

        [Service]
        ExecStart=/opt/bin/apiserver \
        --address=127.0.0.1 \
        --port=8080 \
        --etcd_servers=http://127.0.0.1:4001 \
        --machines=%MINION_IP_ADDRS% \
        --logtostderr=true
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

    - name: scheduler.service
      command: start
      content: |
        [Unit]
        After=apiserver.service
        ConditionFileIsExecutable=/opt/bin/scheduler
        Description=Kubernetes Scheduler
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Wants=apiserver.service

        [Service]
        ExecStart=/opt/bin/scheduler \
        --logtostderr=true \
        --master=127.0.0.1:8080
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

    - name: controller-manager.service
      command: start
      content: |
        [Unit]
        After=etcd.service
        After=download-kubernetes.service
        ConditionFileIsExecutable=/opt/bin/controller-manager
        Description=Kubernetes Controller Manager
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Wants=etcd.service
        Wants=download-kubernetes.service

        [Service]
        ExecStart=/opt/bin/controller-manager \
        --master=127.0.0.1:8080 \
        --logtostderr=true
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

minion.yml

#cloud-config

coreos:
  etcd:
    discovery: http://%ETCD_DISCVERY%:4001/v2/keys/cluster
    addr: $public_ipv4:4001
    peer-addr: $public_ipv4:7001

  fleet:
    public-ip: $public_ipv4

  units:
    - name: etcd.service
      command: start

    - name: fleet.service
      command: start

    - name: flannel.service
      command: start
      content: |
        [Unit]
        Requires=etcd.service
        After=etcd.service

        [Service]
        ExecStart=/opt/bin/flanneld -iface=$public_ipv4

    - name: docker.service
      command: restart
      content: |
        [Unit]
        Description=Docker Application Container Engine
        Documentation=http://docs.docker.io
        Requires=flannel.service
        After=flannel.service

        [Service]
        EnvironmentFile=/run/flannel/subnet.env
        ExecStartPre=-/usr/bin/ip link set dev docker0 down
        ExecStartPre=-/usr/sbin/brctl delbr docker0
        ExecStart=/usr/bin/docker -d -s=btrfs -H fd:// --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
        Restart=on-failure
        RestartSec=5

        [Install]
        WantedBy=multi-user.target

    - name: download-kubernetes.service
      command: start
      content: |
        [Unit]
        After=network-online.target
        Before=kubelet.service
        Before=proxy.service
        Description=Download Kubernetes Binaries
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=network-online.target

        [Service]
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/kubelet
        ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/proxy
        ExecStart=/usr/bin/chmod +x /opt/bin/kubelet
        ExecStart=/usr/bin/chmod +x /opt/bin/proxy
        RemainAfterExit=yes
        Type=oneshot

    - name: kubelet.service
      command: start
      content: |
        [Unit]
        After=etcd.service
        After=download-kubernetes.service
        ConditionFileIsExecutable=/opt/bin/kubelet
        Description=Kubernetes Kubelet
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Wants=etcd.service
        Wants=download-kubernetes.service

        [Service]
        ExecStart=/opt/bin/kubelet \
        --address=0.0.0.0 \
        --port=10250 \
        --hostname_override=$public_ipv4 \
        --etcd_servers=http://127.0.0.1:4001 \
        --logtostderr=true
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

    - name: proxy.service
      command: start
      content: |
        [Unit]
        After=etcd.service
        After=download-kubernetes.service
        ConditionFileIsExecutable=/opt/bin/proxy
        Description=Kubernetes Proxy
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Wants=etcd.service
        Wants=download-kubernetes.service

        [Service]
        ExecStart=/opt/bin/proxy --etcd_servers=http://127.0.0.1:4001 --logtostderr=true
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

redis-master-service.json

{
  "id": "redismaster",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 10000,
  "containerPort": 6379,
  "selector": {
    "name": "redis-master"
  }
}

redis-master.json

{
  "id": "redis-master-2",
  "kind": "Pod",
  "apiVersion": "v1beta1",
  "desiredState": {
    "manifest": {
      "version": "v1beta1",
      "id": "redis-master-2",
      "containers": [{
        "name": "master",
        "image": "dockerfile/redis",
        "ports": [{
          "containerPort": 6379,
          "hostPort": 6379
        }]
      }]
    }
  },
  "labels": {
    "name": "redis-master"
  }
}

redis-slave-controller.json

{
  "id": "redisSlaveController",
  "kind": "ReplicationController",
  "apiVersion": "v1beta1",
  "desiredState": {
    "replicas": 2,
    "replicaSelector": {"name": "redisslave"},
    "podTemplate": {
      "desiredState": {
         "manifest": {
           "version": "v1beta1",
           "id": "redisSlaveController",
           "containers": [{
             "name": "slave",
             "image": "brendanburns/redis-slave",
             "ports": [{"containerPort": 6379, "hostPort": 6380}]
           }]
         }
       },
       "labels": {"name": "redisslave"}
      }},
  "labels": {"name": "redisslave"}
}

redis-slave-service.json

{
  "id": "redisslave",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 10001,
  "containerPort": 6379,
  "labels": {
    "name": "redisslave"
  },
  "selector": {
    "name": "redisslave"
  }
}

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

NUM_MINIONS = 2

BASE_IP_ADDR    = ENV['BASE_IP_ADDR'] || "192.168.12"
ETCD_DISCVERY   = "#{BASE_IP_ADDR}.101"
MASTER_IP_ADDR  = "#{BASE_IP_ADDR}.10"
MINION_IP_ADDRS = NUM_MINIONS.times.collect { |i| BASE_IP_ADDR + ".#{i+11}" }

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "yungsang/coreos-beta"

  config.vm.box_version = "> 1.1.0"

  config.vm.define "discovery" do |discovery|
    discovery.vm.hostname = "discovery"

    discovery.vm.network :private_network, ip: ETCD_DISCVERY

    discovery.vm.provision :file, source: "./discovery.yml", destination: "/tmp/vagrantfile-user-data"

    discovery.vm.provision :shell do |sh|
      sh.privileged = true
      sh.inline = <<-EOT
        mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
      EOT
    end
  end

  config.vm.define "master" do |master|
    master.vm.hostname = "master"

    master.vm.network :forwarded_port, guest: 4001, host: 4001

    master.vm.network :private_network, ip: MASTER_IP_ADDR

    # Install flannel
    master.vm.provision :docker do |d|
      d.run "yungsang/flannel",
        args: "--rm -v /opt/bin:/target",
        auto_assign_name: false, daemonize: false
    end

    master.vm.provision :file, source: "./master.yml", destination: "/tmp/vagrantfile-user-data"

    master.vm.provision :shell do |sh|
      sh.privileged = true
      sh.inline = <<-EOT
        sed -e "s/%ETCD_DISCVERY%/#{ETCD_DISCVERY}/g" -i /tmp/vagrantfile-user-data
        sed -e "s/%MINION_IP_ADDRS%/#{MINION_IP_ADDRS.join(',')}/g" -i /tmp/vagrantfile-user-data

        mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
      EOT
    end
  end

  NUM_MINIONS.times do |i|
    config.vm.define "minion-#{i+1}" do |minion|
      minion.vm.hostname = "minion-#{i+1}"

      minion.vm.network :private_network, ip: MINION_IP_ADDRS[i]

      # Install flannel
      minion.vm.provision :docker do |d|
        d.run "yungsang/flannel",
          args: "--rm -v /opt/bin:/target",
          auto_assign_name: false, daemonize: false
      end

      minion.vm.provision :file, source: "./minion.yml", destination: "/tmp/vagrantfile-user-data"

      minion.vm.provision :shell do |sh|
        sh.privileged = true
        sh.inline = <<-EOT
          sed -e "s/%ETCD_DISCVERY%/#{ETCD_DISCVERY}/g" -i /tmp/vagrantfile-user-data

          mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
        EOT
      end
    end
  end
end

日本語補足
Running Kubernetes Example on CoreOS, Part 2 - Qiita
http://qiita.com/yungsang/items/f286154be7bb6c4ef09f

Running Kubernetes on CoreOS with Rudder - Qiita
http://qiita.com/yungsang/items/530ae3d3277d2fba3343

The master setup file here is out of date. We added a scheduler component that needs to run. Can you please update these instructions? Or better yet, execute our salt setup so that you'll pick up additional changes. Thanks!

Hi @lavalamp,
Thank you for letting me know. I will update it.

Updated with https://github.com/kelseyhightower/kubernetes-coreos/pull/20/files

Updated with flannel.