I wanted to build a home lab to get some experience with how Kubernetes works in a multi-architecture, multi-OS environment. Here's the worklog. I'll probably turn it into a blog later :)
- Setting up a Multi-Arch, Multi OS cluster
I chose this platform because it had 2GB RAM which is the recommended minimum from the Kubeadm docs
-
Log in - found "odroid" dhcp client IP on Mikrotik router
-
Login as root / odroid. Other docs that say odroid/odroid are wrong https://wiki.odroid.com/odroid-xu4/os_images/linux/ubuntu/20170731 (still wrong)
-
Moving root to SSD
- Steps at https://wiki.odroid.com/odroid-xu4/software/building_webserver#make_your_ssd_a_root_partition
- new partition 44da4360-54e4-486d-99cd-74d1d8cfca7f on SATA SSD
- Modified /media/boot/boot.ini, /etc/fstab
- Rsync …
-
Reboot, make sure
mountshows /dev/sda1 mounted at / -
Make a user
useradd -m -s /bin/bash patrick,passwd patrick,usermod -aG sudo patrick -
Log in as new user, make sure
sudoworks -
Lock out root
sudo passwd -l root -
Update everything
apt update ; apt upgrade
I was presented with this:
lqk
x A new boot.ini is installed. x
x Any changes to boot.ini is lost, such as display configuration x
x Persistent custom settings from /media/boot/boot.ini.default x
x have been restored x
x For reference your old boot.ini is saved to x
x /media/boot/boot.ini.old x
tqu
x < OK > x
mqj
Sure enough, it did overwrite it and I had to fix boot.ini. I also tried writing the line to boot.ini.default as it suggested. If I had rebooted, it would have booted from the root filesystem on the microSD card, not the attached SATA SSD. In this case I would have been able to log back in as root/odroid and fix things up. I'll have to remember that's there in case I need it later.
The ODroid device was running warmer than expected at 73C, but not overheating (+85C). This page has steps to enable cpu frequency scaling based on current demand. https://wiki.odroid.com/odroid-xu4/application_note/software/cpufrequtils_cpufreq_govornor
This was enabled by default on the image I used on the Rock64.
I wanted to use the latest Docker CE (18.09) so it would be closer to the version used on Windows Server. The convenience script generally works, but isn't recommended for production. This is pretty dangerous as it's running untrusted code as root.
Running curl -sSL https://get.docker.com | sudo sh will set up the package manager to use Docker's package repo and install for your distro.
$ curl -sSL https://get.docker.com | sudo sh
# Executing docker install script, commit: 4957679
+ sh -c apt-get update -qq >/dev/null
+ sh -c apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sh -c echo "deb [arch=arm64] https://download.docker.com/linux/ubuntu bionic edge" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
+ sh -c docker version
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:52:51 2018
OS/Arch: linux/arm64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:17:01 2018
OS/Arch: linux/arm64
Experimental: false
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:
sudo usermod -aG docker your-user
Remember that you will have to log out and back in for this to take effect!
WARNING: Adding a user to the "docker" group will grant the ability to run
containers which can be used to obtain root privileges on the
docker host.
Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
for more information.
Then I ran sudo usermod -aG docker patrick so I could easily run containers. A log out, then back in, and then tested it with docker run busybox uname -a
Read https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl
Another approach - if you're going to be building and replacing Kubernetes binaries with your own, you could just download them from the v1.13.1 release notes, and copy them to
/usr/local/bin.
I followed Creating a single master cluster from the docs, using Flannel.
The YAML file for Flannel includes a daemonset for most Linux architectures (but not Windows... yet) so it started right up.
$ kubectl -n kube-system get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-flannel-ds-amd64 0 0 0 0 0 beta.kubernetes.io/arch=amd64 7d2h
kube-flannel-ds-arm 1 1 1 1 1 beta.kubernetes.io/arch=arm 7d2h
kube-flannel-ds-arm64 0 0 0 0 0 beta.kubernetes.io/arch=arm64 7d2h
kube-flannel-ds-ppc64le 0 0 0 0 0 beta.kubernetes.io/arch=ppc64le 7d2h
kube-flannel-ds-s390x 0 0 0 0 0 beta.kubernetes.io/arch=s390x 7d2h
kube-proxy 1 1 1 1 1 <none> 7d2hhelm init didn't create a running tipper-deploy deployment. I had to modify it (kubectl edit -n kube-system tiller-deploy) to:
- Tolerate running on a master
- Switch to jessestuart/tiller which is a multiarch image
kubectl get deploy -n kube-system tiller-deploy -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
creationTimestamp: "2018-12-22T21:33:29Z"
generation: 3
labels:
app: helm
name: tiller
name: tiller-deploy
namespace: kube-system
resourceVersion: "2606"
selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy
uid: 3900c67f-0631-11e9-bd4a-001e06328e68
spec:
progressDeadlineSeconds: 2147483647
replicas: 1
revisionHistoryLimit: 2147483647
selector:
matchLabels:
app: helm
name: tiller
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: helm
name: tiller
spec:
automountServiceAccountToken: true
containers:
- env:
- name: TILLER_NAMESPACE
value: kube-system
- name: TILLER_HISTORY_MAX
value: "0"
image: jessestuart/tiller:v2.12.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /liveness
port: 44135
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: tiller
ports:
- containerPort: 44134
name: tiller
protocol: TCP
- containerPort: 44135
name: http
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readiness
port: 44135
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2018-12-22T21:33:29Z"
lastUpdateTime: "2018-12-22T21:33:29Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 3
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Originally I wanted ARM64 / AARCH64, not 32-bit ARM71 / ARMHF. With a bit more research, I found the cheap Rock64 which should work and offers up to 4GB of RAM - double the ODroid HC-1.
@ayufan has a nice set of minimal images ready to run, so I went to the latest release for the Rock64, and downloaded bionic-minimal-rock64-0.7.11-1075-arm64.img.xz. I used Etcher again to set up a 16GB SD Card. I plugged it in, and it booted up within seconds.
The default user+pass is rock64/rock64. I set up a new user (see same steps 5-8 from the ODroid, except I locked out both root and rock64 users) to lock it down.
Swap was also enabled, which isn't supported for Kubernetes.
$ free
total used free shared buff/cache available
Mem: 4083804 133788 2585832 3540 1364184 3914192
Swap: 2041888 0 2041888
patrick@rock64:/$ free
total used free shared buff/cache available
Mem: 4083804 133272 2586352 3540 1364180 3914708
Swap: 2041888 0 2041888
patrick@rock64:/$ mount | grep swap
patrick@rock64:/$ cat /etc/fstab
LABEL=boot /boot/efi vfat defaults,sync 0 0
The Rock64 image has zram-config installed which creates one compressed ramdisk per core. These are then used as "swap" space. That means less used memory pages will be compressed. That would probably be ok in a lab setup, but I disabled it anyway.
- To disable it immediately, run
sudo swapoff -a - Disable it permanently with
sudo apt purge zram-config
You can continue to boot from the SD card at this point, but you can actually boot directly from a USB drive without any SD card at all.
Ayufan's image has a script to install U-Boot in the 128MB SPI flash on the Rock64 itself. If you're booted from the SD card, you can just run sudo /usr/local/sbin/rock64_write_spi_flash.sh.
I still don't know how to control the boot order from this point. It seems to boot from the SD card if it's plugged in and use it as the root. I wasn't able to move the root, but I found another workaround.
Now that U-Boot is installed, it can enumerate USB devices and boot the kernel + root filesystem. If you plug a USB drive (including a SSD/HDD in a USB enclosure) into a PC - you can use Etcher to write the same bionic-minimal-rock64-...-arm64.img.xz file to that drive.
Power off the Rock64, Remove the SD card, and plug in the USB drive. Power it on, and now it will boot directly from USB.
Initially, I decided to leave the boot partition on the SD card, as well as the old root partition so I can boot without the USB SSD attached if I need to repurpose the device later. This didn't work out as expected, but I'll leave the steps here for reference. I recommend the steps in Booting from a USB drive above.
The steps are similar to the ODroid, but the disk layout and boot configuration is different. The ODroid images have U-Boot on the SD card, but the Rock64 uses extlinux.
This guide had most of the steps right, but I copied my exact steps below: https://forum.pine64.org/showthread.php?tid=4971.
- Make a single large GPT volume with
fdisk. Runpto check the volume type,gto change it if needed, andnto create a new partition. Finish withwto save it.
$ sudo fdisk /dev/sda [426/1866]
[sudo] password for patrick:
Welcome to fdisk (util-linux 2.31.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xa7f773cc.
Command (m for help): p
Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: dos
Disk identifier: 0xa7f773cc
Command (m for help): g
Created a new GPT disklabel (GUID: 16C6A535-2FA8-4E40-ABA4-B298D99DD69A).
Command (m for help): p
Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 16C6A535-2FA8-4E40-ABA4-B298D99DD69A
Command (m for help): n
Partition number (1-128, default 1):
First sector (65535-500118158, default 65535):
Last sector, +sectors or +size{K,M,G,T,P} (65535-500118158, default 500118158):
Created a new partition 1 of type 'Linux filesystem' and of size 238.5 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.- Format it ext4, then mount it to
/mnt/ssd
$ sudo mkfs.ext4 /dev/sda1
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 62506578 4k blocks and 15630336 inodes
Filesystem UUID: 8d305b55-4638-4c54-87b9-e39df4c04279
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mkdir -p /mnt/ssd
$ sudo mount /dev/sda1 /mnt/ssd- Copy the existing root over to the SSD
$ sudo rsync -axv /- Get the UUID for the new root partition.
$ lsblk -f /dev/sda
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
`-sda1 ext4 8d305b55-4638-4c54-87b9-e39df4c04279 /mnt/ssd-
Modify
/boot/extlinux.confand changeroot=toroot=UUID=<UUID from previous step> -
Unmount the new filesystem
sudo umount /dev/sda1 -
sudo reboot
After the reboot, double check that it worked. sda1 should be mounted to /
$ lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
`-sda1 ext4 8d305b55-4638-4c54-87b9-e39df4c04279 /
mtdblock0
mmcblk1
|-mmcblk1p1
|-mmcblk1p2
|-mmcblk1p3
|-mmcblk1p4
|-mmcblk1p5
|-mmcblk1p6 vfat boot 226D-5878 /boot/efi
`-mmcblk1p7 ext4 linux-root d6d72015-76f9-4662-9bb9-6effe2437f69
zram0 [SWAP]
zram1 [SWAP]
zram2 [SWAP]
zram3 [SWAP]The same doc also shows how to join a node to the existing cluster.
I can't get another computer delivered by Tardis, so my kubeadm token expired by the time the next machine arrived.
On the existing cluster leader, I ran
kubeadm token createto create a new tokenopenssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'to get the certip addr show eth0to double check the IP
Then on the new node, I ran
kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
And now it's in the cluster
$ kubectl get node
NAME STATUS ROLES AGE VERSION
odroid Ready master 7d3h v1.13.1
rock64 NotReady <none> 40s v1.13.1After a few seconds, the status will change to Ready.
-
Get the existing Flannel configuration with
kubectl get configmap -n kube-system kube-flannel-cfg -o yaml > kube-flannel-cfg.yaml -
Modify it with
vim kube-flannel-cfg.yaml
Change the contents of net-conf.json so that the Backend type is host-gw
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "host-gw"
}
}-
Replace the old configmap with the new one
kubectl replace configmap -n kube-system kube-flannel-cfg -f kube-flannel-cfg.yaml -
Find and kill all of the Flannel DaemonSet pods with
kubectl get pod -n kube-systemandkubectl delete pod -n kube-system kube-flannel-ds-...
Here's some stuff I should link to when I make this a blog:
- Some explanation of why Linux, golang, and Docker have different names for the ARM architecture variants
- How to use ContainerD instead of Docker CE