Skip to content

Instantly share code, notes, and snippets.

@YungSang
Last active April 3, 2017 09:10
Show Gist options
  • Save YungSang/6177b69f1754f0590dbe to your computer and use it in GitHub Desktop.
Save YungSang/6177b69f1754f0590dbe to your computer and use it in GitHub Desktop.
Running Kubernetes Example on CoreOS, Part 2 with flannel (formerly Rudder)

Running Kubernetes Example on CoreOS, Part 2 with flannel (formerly Rudder)

Step Zero: Build up a Kubernetes cluster

$ git clone https://gist.github.com/6177b69f1754f0590dbe.git
$ vagrant up

It will boot up one for etcd discovery, one master and two minion servers (minion-x).

Setup an SSH tunnel

Setup an SSH tunnel to the Kubernetes API Server in order to use kubecfg on your local machine.

$ curl -OL http://storage.googleapis.com/kubernetes/darwin/kubecfg
$ chmod +x kubecfg
$ mv kubecfg /usr/local/bin
$ kubecfg --version
Kubernetes v0.2-dev
$ vagrant ssh-config master > ssh.config
$ ssh -f -nNT -L 8080:127.0.0.1:8080 -F ssh.config master
$ kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------

Cf.) https://github.com/kelseyhightower/kubernetes-coreos#running-commands-remotely

Step One: Turn up the redis master

$ kubecfg -c redis-master.json create pods
I0919 21:28:53.758488 29403 request.go:292] Waiting for completion of /operations/1
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    /                   name=redis-master   Waiting

$ kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    192.168.12.11/      name=redis-master   Waiting

$ kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
redis-master-2      dockerfile/redis    192.168.12.11/      name=redis-master   Running

Step Two: Turn up the master service

$ kubecfg -c redis-master-service.json create services
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000

$ kubecfg list services
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000

Step Three: Turn up the replicated slave pods

$ kubecfg -c redis-slave-controller.json create replicationControllers
I0919 21:35:28.725306 29856 request.go:292] Waiting for completion of /operations/4
ID                     Image(s)                   Selector            Replicas
----------             ----------                 ----------          ----------
redisSlaveController   brendanburns/redis-slave   name=redisslave     2

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Waiting
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Waiting

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Running
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Running

Step Four: Create the redis slave service

$ kubecfg -c redis-slave-service.json create services
I0919 21:46:52.782728 30375 request.go:292] Waiting for completion of /operations/9
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redisslave          name=redisslave     name=redisslave     10001

$ kubecfg list services
ID                  Labels              Selector            Port
----------          ----------          ----------          ----------
redismaster                             name=redis-master   10000
redisslave          name=redisslave     name=redisslave     10001

Step Five: Create the frontend pod

$ kubecfg -c frontend-controller.json create replicationControllers
I0919 21:47:38.445604 30429 request.go:292] Waiting for completion of /operations/10
ID                   Image(s)                 Selector            Replicas
----------           ----------               ----------          ----------
frontendController   brendanburns/php-redis   name=frontend       2

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Running
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Running
3f3fc79a-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.12/      name=frontend,replicationController=frontendController       Waiting
3f3ff5ff-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.11/      name=frontend,replicationController=frontendController       Waiting
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running

$ kubecfg list pods
ID                                     Image(s)                   Host                Labels                                                       Status
----------                             ----------                 ----------          ----------                                                   ----------
redis-master-2                         dockerfile/redis           192.168.12.11/      name=redis-master                                            Running
8c46c480-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.12/      name=redisslave,replicationController=redisSlaveController   Running
8c472246-407f-11e4-a8ea-0800271cb107   brendanburns/redis-slave   192.168.12.11/      name=redisslave,replicationController=redisSlaveController   Running
3f3fc79a-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.12/      name=frontend,replicationController=frontendController       Running
3f3ff5ff-4081-11e4-a8ea-0800271cb107   brendanburns/php-redis     192.168.12.11/      name=frontend,replicationController=frontendController       Running

$ open http://192.168.12.11:8000
$ open http://192.168.12.12:8000
#cloud-config
coreos:
etcd:
addr: $public_ipv4:4001
peer-addr: $public_ipv4:7001
units:
- name: etcd.service
command: start
{
"id": "frontendController",
"kind": "ReplicationController",
"apiVersion": "v1beta1",
"desiredState": {
"replicas": 2,
"replicaSelector": {"name": "frontend"},
"podTemplate": {
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "frontendController",
"containers": [{
"name": "php-redis",
"image": "brendanburns/php-redis",
"ports": [{"containerPort": 80, "hostPort": 8000}]
}]
}
},
"labels": {"name": "frontend"}
}},
"labels": {"name": "frontend"}
}
#cloud-config
coreos:
etcd:
discovery: http://%ETCD_DISCVERY%:4001/v2/keys/cluster
addr: $public_ipv4:4001
peer-addr: $public_ipv4:7001
units:
- name: etcd.service
command: start
- name: flannel.service
command: start
content: |
[Unit]
Requires=etcd.service
After=etcd.service
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.100.0.0/16"}'
ExecStart=/opt/bin/flanneld -iface=$public_ipv4
[Install]
WantedBy=multi-user.target
- name: docker.service
command: restart
content: |
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
Requires=flannel.service
After=flannel.service
[Service]
EnvironmentFile=/run/flannel/subnet.env
ExecStartPre=-/usr/bin/ip link set dev docker0 down
ExecStartPre=-/usr/sbin/brctl delbr docker0
ExecStart=/usr/bin/docker -d -s=btrfs -H fd:// --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
- name: download-kubernetes.service
command: start
content: |
[Unit]
After=network-online.target
Before=apiserver.service
Before=controller-manager.service
Description=Download Kubernetes Binaries
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=network-online.target
[Service]
ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/apiserver
ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/controller-manager
ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/kubecfg
ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/scheduler
ExecStart=/usr/bin/chmod +x /opt/bin/apiserver
ExecStart=/usr/bin/chmod +x /opt/bin/controller-manager
ExecStart=/usr/bin/chmod +x /opt/bin/kubecfg
ExecStart=/usr/bin/chmod +x /opt/bin/scheduler
RemainAfterExit=yes
Type=oneshot
- name: apiserver.service
command: start
content: |
[Unit]
After=etcd.service
After=download-kubernetes.service
ConditionFileIsExecutable=/opt/bin/apiserver
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Wants=etcd.service
Wants=download-kubernetes.service
[Service]
ExecStart=/opt/bin/apiserver \
--address=127.0.0.1 \
--port=8080 \
--etcd_servers=http://127.0.0.1:4001 \
--machines=%MINION_IP_ADDRS% \
--logtostderr=true
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
- name: scheduler.service
command: start
content: |
[Unit]
After=apiserver.service
ConditionFileIsExecutable=/opt/bin/scheduler
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Wants=apiserver.service
[Service]
ExecStart=/opt/bin/scheduler \
--logtostderr=true \
--master=127.0.0.1:8080
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
- name: controller-manager.service
command: start
content: |
[Unit]
After=etcd.service
After=download-kubernetes.service
ConditionFileIsExecutable=/opt/bin/controller-manager
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Wants=etcd.service
Wants=download-kubernetes.service
[Service]
ExecStart=/opt/bin/controller-manager \
--master=127.0.0.1:8080 \
--logtostderr=true
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
#cloud-config
coreos:
etcd:
discovery: http://%ETCD_DISCVERY%:4001/v2/keys/cluster
addr: $public_ipv4:4001
peer-addr: $public_ipv4:7001
fleet:
public-ip: $public_ipv4
units:
- name: etcd.service
command: start
- name: fleet.service
command: start
- name: flannel.service
command: start
content: |
[Unit]
Requires=etcd.service
After=etcd.service
[Service]
ExecStart=/opt/bin/flanneld -iface=$public_ipv4
- name: docker.service
command: restart
content: |
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
Requires=flannel.service
After=flannel.service
[Service]
EnvironmentFile=/run/flannel/subnet.env
ExecStartPre=-/usr/bin/ip link set dev docker0 down
ExecStartPre=-/usr/sbin/brctl delbr docker0
ExecStart=/usr/bin/docker -d -s=btrfs -H fd:// --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
- name: download-kubernetes.service
command: start
content: |
[Unit]
After=network-online.target
Before=kubelet.service
Before=proxy.service
Description=Download Kubernetes Binaries
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Requires=network-online.target
[Service]
ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/kubelet
ExecStart=/usr/bin/wget -N -P /opt/bin http://storage.googleapis.com/kubernetes/proxy
ExecStart=/usr/bin/chmod +x /opt/bin/kubelet
ExecStart=/usr/bin/chmod +x /opt/bin/proxy
RemainAfterExit=yes
Type=oneshot
- name: kubelet.service
command: start
content: |
[Unit]
After=etcd.service
After=download-kubernetes.service
ConditionFileIsExecutable=/opt/bin/kubelet
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Wants=etcd.service
Wants=download-kubernetes.service
[Service]
ExecStart=/opt/bin/kubelet \
--address=0.0.0.0 \
--port=10250 \
--hostname_override=$public_ipv4 \
--etcd_servers=http://127.0.0.1:4001 \
--logtostderr=true
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
- name: proxy.service
command: start
content: |
[Unit]
After=etcd.service
After=download-kubernetes.service
ConditionFileIsExecutable=/opt/bin/proxy
Description=Kubernetes Proxy
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Wants=etcd.service
Wants=download-kubernetes.service
[Service]
ExecStart=/opt/bin/proxy --etcd_servers=http://127.0.0.1:4001 --logtostderr=true
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
{
"id": "redismaster",
"kind": "Service",
"apiVersion": "v1beta1",
"port": 10000,
"containerPort": 6379,
"selector": {
"name": "redis-master"
}
}
{
"id": "redis-master-2",
"kind": "Pod",
"apiVersion": "v1beta1",
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "redis-master-2",
"containers": [{
"name": "master",
"image": "dockerfile/redis",
"ports": [{
"containerPort": 6379,
"hostPort": 6379
}]
}]
}
},
"labels": {
"name": "redis-master"
}
}
{
"id": "redisSlaveController",
"kind": "ReplicationController",
"apiVersion": "v1beta1",
"desiredState": {
"replicas": 2,
"replicaSelector": {"name": "redisslave"},
"podTemplate": {
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "redisSlaveController",
"containers": [{
"name": "slave",
"image": "brendanburns/redis-slave",
"ports": [{"containerPort": 6379, "hostPort": 6380}]
}]
}
},
"labels": {"name": "redisslave"}
}},
"labels": {"name": "redisslave"}
}
{
"id": "redisslave",
"kind": "Service",
"apiVersion": "v1beta1",
"port": 10001,
"containerPort": 6379,
"labels": {
"name": "redisslave"
},
"selector": {
"name": "redisslave"
}
}
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
NUM_MINIONS = 2
BASE_IP_ADDR = ENV['BASE_IP_ADDR'] || "192.168.12"
ETCD_DISCVERY = "#{BASE_IP_ADDR}.101"
MASTER_IP_ADDR = "#{BASE_IP_ADDR}.10"
MINION_IP_ADDRS = NUM_MINIONS.times.collect { |i| BASE_IP_ADDR + ".#{i+11}" }
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "yungsang/coreos-beta"
config.vm.box_version = "> 1.1.0"
config.vm.define "discovery" do |discovery|
discovery.vm.hostname = "discovery"
discovery.vm.network :private_network, ip: ETCD_DISCVERY
discovery.vm.provision :file, source: "./discovery.yml", destination: "/tmp/vagrantfile-user-data"
discovery.vm.provision :shell do |sh|
sh.privileged = true
sh.inline = <<-EOT
mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
EOT
end
end
config.vm.define "master" do |master|
master.vm.hostname = "master"
master.vm.network :forwarded_port, guest: 4001, host: 4001
master.vm.network :private_network, ip: MASTER_IP_ADDR
# Install flannel
master.vm.provision :docker do |d|
d.run "yungsang/flannel",
args: "--rm -v /opt/bin:/target",
auto_assign_name: false, daemonize: false
end
master.vm.provision :file, source: "./master.yml", destination: "/tmp/vagrantfile-user-data"
master.vm.provision :shell do |sh|
sh.privileged = true
sh.inline = <<-EOT
sed -e "s/%ETCD_DISCVERY%/#{ETCD_DISCVERY}/g" -i /tmp/vagrantfile-user-data
sed -e "s/%MINION_IP_ADDRS%/#{MINION_IP_ADDRS.join(',')}/g" -i /tmp/vagrantfile-user-data
mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
EOT
end
end
NUM_MINIONS.times do |i|
config.vm.define "minion-#{i+1}" do |minion|
minion.vm.hostname = "minion-#{i+1}"
minion.vm.network :private_network, ip: MINION_IP_ADDRS[i]
# Install flannel
minion.vm.provision :docker do |d|
d.run "yungsang/flannel",
args: "--rm -v /opt/bin:/target",
auto_assign_name: false, daemonize: false
end
minion.vm.provision :file, source: "./minion.yml", destination: "/tmp/vagrantfile-user-data"
minion.vm.provision :shell do |sh|
sh.privileged = true
sh.inline = <<-EOT
sed -e "s/%ETCD_DISCVERY%/#{ETCD_DISCVERY}/g" -i /tmp/vagrantfile-user-data
mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/
EOT
end
end
end
end
@YungSang
Copy link
Author

YungSang commented Aug 1, 2014

日本語補足
Running Kubernetes Example on CoreOS, Part 2 - Qiita
http://qiita.com/yungsang/items/f286154be7bb6c4ef09f

@YungSang
Copy link
Author

Running Kubernetes on CoreOS with Rudder - Qiita
http://qiita.com/yungsang/items/530ae3d3277d2fba3343

@lavalamp
Copy link

The master setup file here is out of date. We added a scheduler component that needs to run. Can you please update these instructions? Or better yet, execute our salt setup so that you'll pick up additional changes. Thanks!

@YungSang
Copy link
Author

Hi @lavalamp,
Thank you for letting me know. I will update it.

@YungSang
Copy link
Author

@YungSang
Copy link
Author

Updated with flannel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment