Offline Kubeadm install

kubeadm-install-offline.md

Install Docker

• Download all dependencies from by following the documentation https://docs.docker.com/engine/installation/linux/docker-ce/centos/#install-using-the-repository

yumdownloader --resolve yum-utils device-mapper-persistent-data lvm2
yumdownloader --resolve docker-ce

• Copy to master node and install afterwards:

yum install -y *.rpm

On master and nodes

Pull images form internet access laptop

docker pull gcr.io/google_containers/kube-apiserver-amd64:v1.6.0
docker pull gcr.io/google_containers/kube-controller-manager-amd64:v1.6.0
docker pull gcr.io/google_containers/kube-proxy-amd64:v1.6.0
docker pull gcr.io/google_containers/kube-scheduler-amd64:v1.6.0
docker pull gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0
docker pull gcr.io/google_containers/etcd-amd64:3.0.17
docker pull gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1
docker pull gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1
docker pull gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1
docker pull quay.io/coreos/flannel:v0.7.1-amd64
docker pull gcr.io/google_containers/pause-amd64:3.0
docker pull gcr.io/kubernetes-helm/tiller:v2.3.0
docker pull registry
docker pull busybox
docker pull weaveworks/weave-npc:2.0.4
docker pull weaveworks/weave-kube:2.0.4
* docker pull gcr.io/google-containers/kube-addon-manager:v6.1
* docker pull gcr.io/google_containers/exechealthz-amd64:1.2
* docker pull gcr.io/google_containers/kube-discovery-amd64:1.0

Save docker images to tarball

docker save gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 > kube-apiserver-amd64:v1.6.0.tar
docker save gcr.io/google_containers/kube-controller-manager-amd64:v1.6.0 > kube-controller-manager-amd64:v1.6.0.tar
docker save gcr.io/google_containers/kube-proxy-amd64:v1.6.0 > kube-proxy-amd64:v1.6.0.tar
docker save gcr.io/google_containers/kube-scheduler-amd64:v1.6.0 > kube-scheduler-amd64:v1.6.0.tar
docker save gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0 > kubernetes-dashboard-amd64:v1.6.0.tar
docker save gcr.io/google_containers/etcd-amd64:3.0.17 > etcd-amd64:3.0.17.tar
docker save gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1 > k8s-dns-kube-dns-amd64:1.14.1.tar
docker save gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1 > k8s-dns-dnsmasq-nanny-amd64:1.14.1.tar
docker save gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1 > k8s-dns-sidecar-amd64:1.14.1.tar
docker save quay.io/coreos/flannel:v0.7.1-amd64 > flannel:v0.7.1-amd64.tar
docker save gcr.io/google_containers/pause-amd64:3.0 > gcr.io/google_containers/pause-amd64:3.0.tar
docker save gcr.io/kubernetes-helm/tiller:v2.3.0 > gcr.io/kubernetes-helm/tiller:v2.3.0.tar
docker save registry > registry.tar
docker save busybox > busybox.tar
docker save weaveworks/weave-npc:2.0.4 > weave-npc:2.0.4.tar
docker save weaveworks/weave-kube:2.0.4 > weave-kube:2.0.4.tar
* docker save gcr.io/google-containers/kube-addon-manager:v6.1 > kube-addon-manager:v6.1.tar
* docker save gcr.io/google_containers/exechealthz-amd64:1.2 > exechealthz-amd64:1.2.tar
* docker save gcr.io/google_containers/kube-discovery-amd64:1.0 > kube-discovery-amd64:1.0.tar

Copy docker images tarball to the remote server

scp <folder_with_images>/*.tar <user>@<server>:<path>/<to>/<remote>/<folder>

Ensure docker is started

systemctl status docker

In case docker is not started

systemctl enable docker && systemctl start docker

Load docker images on remote server

docker load < kube-apiserver-amd64:v1.6.0.tar
docker load < kube-controller-manager-amd64:v1.6.0.tar
docker load <  kube-proxy-amd64:v1.6.0.tar
docker load <  kube-scheduler-amd64:v1.6.0.tar
docker load < kubernetes-dashboard-amd64:v1.6.0.tar
docker load <  etcd-amd64:3.0.17.tar
docker load <  k8s-dns-kube-dns-amd64:1.14.1.tar
docker load <  k8s-dns-dnsmasq-nanny-amd64:1.14.1.tar
docker load <  k8s-dns-sidecar-amd64:1.14.1.tar
docker load < flannel:v0.7.1-amd64.tar
docker load <  gcr.io/google_containers/pause-amd64:3.0.tar
docker load <  gcr.io/kubernetes-helm/tiller:v2.3.0.tar
docker load < registry.tar
docker load < busybox.tar
docker load < weave-npc:2.0.4.tar
docker load < weave-kube:2.0.4.tar
* docker load < kube-addon-manager:v6.1.tar
* docker load <  exechealthz-amd64:1.2.tar
* docker load <  kube-discovery-amd64:1.0.tar

For Centos7, enable sysctl configuration

edit /etc/sysctl.conf

vi /etc/sysctl.conf

modify

net.ipv4.ip_forward = 0 to net.ipv4.ip_forward = 1

Append

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

Reload properties

sysctl -p

Download kubernetes rpms

yumdownloader --resolve ebtables ethtool iproute iptables libmnl libnetfilter_conntrack libnfnetlink socat tcp_wrappers-libs
wget https://packages.cloud.google.com/yum/pool/e6aef7b2b7d9e5bd4db1e5747ebbc9f1f97bbfb8c7817ad68028565ca263a672-kubectl-1.6.0.x86_64.rpm
wget https://packages.cloud.google.com/yum/pool/af8567f1ba6f8dc1d43b60702d45c02aca88607b0e721d76897e70f6a6e53115-kubelet-1.6.0.x86_64.rpm
wget https://packages.cloud.google.com/yum/pool/e7a4403227dd24036f3b0615663a371c4e07a95be5fee53505e647fd8ae58aa6-kubernetes-cni-0.5.1.x86_64.rpm
wget https://packages.cloud.google.com/yum/pool/5116fa4b73c700823cfc76b3cedff6622f2fbd0a3d2fa09bce6d93329771e291-kubeadm-1.6.0.x86_64.rpm

Copy kubernetes rpms to the remote server

scp <folder_with_rpms>/*.rpm <user>@<server>:<path>/<to>/<remote>/<folder>

Install kubernetes tools

yum install -y *.rpm
systemctl enable kubelet && systemctl start kubelet

On master

Kubeadm installation

Follow instruction from https://kubernetes.io/docs/getting-started-guides/kubeadm/ (Starting from (2/4) Initializing your master)

initalization

kubeadm init

...
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[init] Using Kubernetes version: v1.5.1
[tokens] Generated token: "064158.548b9ddb1d3fad3e"
[certificates] Generated Certificate Authority key and certificate.
[certificates] Generated API Server key and certificate
[certificates] Generated Service Account signing keys
[certificates] Created keys and certificates in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 61.317580 seconds
[apiclient] Waiting for at least one node to register and become ready
[apiclient] First node is ready after 6.556101 seconds
[apiclient] Creating a test deployment
[apiclient] Test deployment succeeded
[token-discovery] Created the kube-discovery deployment, waiting for it to become ready
[token-discovery] kube-discovery is ready after 6.020980 seconds
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
    http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node:

kubeadm join --token=<token> <master-ip>
# Copy the previous line
...

Network configuration

Download weave plugin configuration (with internet)

wget https://git.io/weave-kube
mv weave-kube weave-kube.yml

Copy weave plugin configuration

scp <folder_with_weave_yml>/weave-kube.yml <user>@<server>:<path>/<to>/<remote>/<folder>

Apply plugin

kubectl apply -f <folder_with_weave_yml>/weave-kube.yml

Fix kube-proxy problem

kubectl -n kube-system get ds -l "component=kube-proxy" -o json | jq ".items[0].spec.template.spec.containers[0].command |= .+ [\"--proxy-mode=userspace\"]" | kubectl apply -f - && kubectl -n kube-system delete pods -l "component=kube-proxy" 

On Node

kubeadm join --token

Trick and tips

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.7.0/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Question - it appears that etcd is failing with a timeout. Do you have any recommendations for offline mode initialization of three manager/control plane nodes via kubeadm join --control-plane