custom hyperkube.md

Mac users

1. First, install GNU tar: brew install --with-default-names gnu-tar
2. Ensure at least 4GB RAM available to Docker for Mac

• https://docs.docker.com/docker-for-mac/#advanced

clone the repo

go get k8s.io/kubernetes && cd $GOPATH/src/k8s.io/kubernetes

(commit changes to branch)

Build Kubernetes binaries and hyperkube image

Build an arch/amd64 only version of Kubernetes binaries

In the Kubernetes repo:

bash build/run.sh make cross KUBE_FASTBUILD=true ARCH=amd64

Without the KUBE_FASTBUILD you will cross compile a matrix of platforms and waste 45mins of your time. Should take 10-15 minutes when compiling for linux/amd64.

Create Docker image (hyperkube)

ACS runs Kubernetes via a single container image called hyperkube, as opposed to individual binaries on the host VMs.

The following will put the compiled k8s binaries into a Docker image.

Build image

cd cluster/images/hyperkube/
make VERSION=arm-backoff-spike ARCH=amd64 REGISTRY=seanknox
# it downloads some base images, and creates your release image
hyperkube$ docker images
REPOSITORY                                   TAG                           IMAGE ID            CREATED             SIZE
seanknox/hyperkube-amd64                                 arm-backoff-spike                          44c30ac8c8a2        17 minutes ago      608MB

Get a build error?

$ make build VERSION=arm-backoff-spike ARCH=amd64 REGISTRY=seanknox
cp -r ./* /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP
mkdir -p /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP/cni-bin
cp ../../../_output/dockerized/bin/linux/amd64/hyperkube /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP
chmod a+rx /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP/hyperkube
cd /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP && sed -i.back "s|ARCH||g" Dockerfile
cd /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP && sed -i.back "s|BASEIMAGE|debian:jessie|g" Dockerfile
cd /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP && sed -i.back "s|CACHEBUST|B3B345E5-57E4-4A1E-954F-E9A80EAFC9DE|g" Dockerfile
# When building "normally" for amd64, remove the whole line, it has no part in the amd64 image
cd /var/folders/w0/20ws9w_s5rzdym83kjq423g80000gp/T/hyperkubeXXXXXX.OoE9jbhP && sed -i "" "/CROSS_BUILD_/d" Dockerfile
sed: can't read /CROSS_BUILD_/d: No such file or directory
make: *** [build] Error 2

...means you're using gnu-sed on MacOS. You'll need to ensure the default MacOS bin /usr/bin/sed is foremost in your $PATH. I uninstalled it (homebrew uninstall gnu-sed).

Push image to registry

docker push seanknox/hyperkube-amd64

upgrade_acs_k8s_masters.sh

#!/bin/sh
export OLD_IMAGE="gcr.io/google_containers/hyperkube-amd64:v1.6.4"
export NEW_IMAGE="seanknox/hyperkube-amd64:arm-backoff-spike"

# these lines are for older acs-engine cluster or clusters provisioned via `az acs`
sed -i -e "s@${OLD_IMAGE}@${NEW_IMAGE}@g" /etc/systemd/system/kubelet.service
sed -i -e "s@--config=@--pod-manifest-path=@g" /etc/systemd/system/kubelet.service

sed -i -e "s@${OLD_IMAGE}@${NEW_IMAGE}@g" /etc/default/kubelet
sed -i -e "s@${OLD_IMAGE}@${NEW_IMAGE}@g" /etc/kubernetes/manifests/*

systemctl daemon-reload
systemctl restart kubelet

upgrade_node.sh

#!/bin/sh
export OLD_IMAGE="gcrio.azureedge.net/google_containers/hyperkube-amd64:v1.6.2"
export NEW_IMAGE="seanknox/hyperkube-amd64:arm-backoff-spike"
# these lines are for older acs-engine cluster or clusters provisioned via `az acs`
sed -i -e "s@${OLD_IMAGE}@${NEW_IMAGE}@g" /etc/systemd/system/kubelet.service
sed -i -e "s@--config=@--pod-manifest-path=@g" /etc/systemd/system/kubelet.service

# for current acs-engine deploys
sed -i -e "s@${OLD_IMAGE}@${NEW_IMAGE}@g" /etc/default/kubelet

systemctl daemon-reload
systemctl restart kubelet

I think this line does not appear to do anything:

sed -i -e "s@${OLD_IMAGE}@${NEW_IMAGE}@g" /etc/systemd/system/kubelet.service

And this line we need in the master upgrade script:

sed -i -e "s@${OLD_IMAGE}@${NEW_IMAGE}@g" /etc/default/kubelet

Thanks! will update.

For posterity, my flow:

Build new image

From the kubernetes repo root:

$ export DOCKER_WHOAMI=jackfrancis
$ export HYPERKUBE_DEV_VERSION=PR-46660.05
$ bash build/run.sh make cross KUBE_FASTBUILD=true ARCH=amd64 && cd cluster/images/hyperkube/ && make VERSION=$HYPERKUBE_DEV_VERSION ARCH=amd64 REGISTRY=$DOCKER_WHOAMI && docker push $DOCKER_WHOAMI/hyperkube-amd64:$HYPERKUBE_DEV_VERSION && cd ../../..

Build new cluster

From the acs-engine repo root:

$ grep 'CUSTOM_HYPERKUBE_SPEC' test/user.env
export CUSTOM_HYPERKUBE_SPEC=jackfrancis/hyperkube-amd64:PR-46660.05
$ make build && ./test/deploy.sh examples/kubernetes.json &

☕