Instructions I've been cobbling together to potentially build docker images in a VM instead of the Docker.app

docker vm instructions.md

Inspiration:

• https://medium.com/@gourneau_38759/docker-engine-with-multipass-on-macos-dc44ff09ffb2
• https://zhimin-wen.medium.com/running-k3s-with-multipass-on-mac-fbd559966f7c
• https://blog.mikesir87.io/2019/08/using-ssh-connections-in-docker-contexts/
• https://computingforgeeks.com/install-kubernetes-on-ubuntu-using-k3s/
• https://andreipope.github.io/tutorials/create-a-cluster-with-multipass-and-k3s
• https://levelup.gitconnected.com/kubernetes-cluster-with-k3s-and-multipass-7532361affa3

Install multipass and docker.

brew install docker
brew install docker-credential-helper # This allows us to do `docker login cf-registry.nr-ops.net`
brew install --cask multipass
brew install kubectl

This will create a new "cloud-init" file for the master and copy in your public key from ~/.ssh/id_rsa.pub (or ~/.ssh/id_ed25519)

export PUBLIC_KEY=$(cat ~/.ssh/id_*.pub)

cat <<EOF > ~/docker.yaml
---  
users:  
  - name: ubuntu  
    sudo: ALL=(ALL) NOPASSWD:ALL  
    ssh_authorized_keys:  
      - $PUBLIC_KEY

package_update: true  
packages:  
  - docker  
  - avahi-daemon  
  - apt-transport-https  
  - ca-certificates  
  - curl  
  - gnupg  
  - lsb-release

runcmd:  
  - sudo curl -fsSL https://get.docker.com | sudo bash 
  - sudo systemctl enable docker  
  - sudo systemctl enable -s HUP ssh  
  - sudo groupadd docker  
  - sudo usermod -aG docker ubuntu
EOF

This will launch a new VM (this takes a while) using the file we created above:

multipass launch --verbose --cpus 2 --mem 2G --disk 5G --name docker --cloud-init ~/docker.yaml 22.04

Mandatory Now SSH into the new VM to get past the fingerprint, but also to check some things out:

ssh ubuntu@docker.local
The authenticity of host 'docker.local (192.168.64.2)' can't be established.
ED25519 key fingerprint is SHA256:dhL//UV2I/YTKfHwIqUBhBn9293LaY2jLGkSvKvuS+I.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'docker.local' (ED25519) to the list of known hosts.

docker info

exit

Check to make sure it's working outside the VM too (the hard way):

DOCKER_HOST="ssh://ubuntu@docker.local" docker ps

Now let's make it easy to connect to that dockerd by setting the default context:

docker context create ssh-box --docker "host=ssh://ubuntu@docker.local"
docker context use ssh-box
docker ps

Adding in Kubernetes

Let's create a master container

export PUBLIC_KEY=$(cat ~/.ssh/id_*.pub)

cat <<EOF > ~/k3s-master.yaml
---  
users:  
  - name: ubuntu  
    sudo: ALL=(ALL) NOPASSWD:ALL  
    ssh_authorized_keys:  
      - $PUBLIC_KEY

package_update: true  
packages:  
  - docker  
  - avahi-daemon  
  - apt-transport-https  
  - ca-certificates  
  - curl  
  - gnupg  
  - lsb-release

runcmd:  
  - sudo curl -fsSL https://get.docker.com | sudo bash 
  - sudo systemctl enable docker  
  - sudo systemctl enable -s HUP ssh  
  - sudo groupadd docker  
  - sudo usermod -aG docker ubuntu
  - curl -sfL https://get.k3s.io | sh -s - --docker
  - sudo ufw allow 22/tcp
  - sudo ufw allow 443/tcp
  - sudo ufw allow 6443/tcp
EOF

This will launch a new VM (this takes a while) using the file we created above:

multipass launch --verbose --cpus 2 --mem 2G --disk 5G --name k3s-master --cloud-init ~/k3s-master.yaml 22.04

Mandatory Now SSH into the new VM to get past the fingerprint, but also to check some things out:

ssh ubuntu@k3s-master.local
The authenticity of host 'k3s-master.local (192.168.64.2)' can't be established.
ED25519 key fingerprint is SHA256:dhL//UV2I/YTKfHwIqUBhBn9293LaY2jLGkSvKvuS+I.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'k3s-master.local' (ED25519) to the list of known hosts.

docker info
kubectl get nodes

exit

Start by getting some configuration from the master

TOKEN=$(multipass exec k3s-master sudo cat /var/lib/rancher/k3s/server/node-token)
IP=$(multipass info k3s-master | grep IPv4 | awk '{print $2}')

Like the master, create a new cloud-init file for the workers

export PUBLIC_KEY=$(cat ~/.ssh/id_*.pub)

cat <<EOF > ~/worker.yaml
---  
users:  
  - name: ubuntu  
    sudo: ALL=(ALL) NOPASSWD:ALL  
    ssh_authorized_keys:  
      - $PUBLIC_KEY

package_update: true  
packages:  
  - docker  
  - avahi-daemon  
  - apt-transport-https  
  - ca-certificates  
  - curl  
  - gnupg  
  - lsb-release

runcmd:  
  - sudo curl -fsSL https://get.docker.com | sudo bash 
  - sudo systemctl enable docker  
  - sudo systemctl enable -s HUP ssh  
  - sudo groupadd docker  
  - sudo usermod -aG docker ubuntu
  - sudo curl -sfL https://get.k3s.io | K3S_URL="https://$IP:6443" K3S_TOKEN="$TOKEN" sh -s - --docker
EOF

Now launch the two workers and configure them to talk with the master

multipass launch --verbose --cpus 1 --mem 1G --disk 4G --name k3s-worker-1 --cloud-init ~/worker.yaml 22.04
multipass launch --verbose --cpus 1 --mem 1G --disk 4G --name k3s-worker-2 --cloud-init ~/worker.yaml 22.04

ssh ubuntu@k3s-master.local

kubectl get nodes
kubectl get pods --all-namespaces

exit

Lastly, let's make it easy to connect to the k3s cluster from our laptop

ssh ubuntu@k3s-master.local

mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown ubuntu:ubuntu ~/.kube/config

exit

multipass copy-files docker:/etc/rancher/k3s/k3s.yaml ${HOME}/.kube/k3s.yaml
sed -ie s,https://127.0.0.1:6443,${IP},g ${HOME}/.kube/k3s.yaml
kubectl --kubeconfig=${HOME}/.kube/k3s.yaml get nodes
export KUBECONFIG=${HOME}/.kube/k3s.yaml
kubectl get nodes