Run Minikube in LXD

setup.md

Running Minikube via LXD

I make some assumptions, and make no claims in how well supported this is or ever will be. I wanted to avoid using VMs because i've been working in containers for the last half decade. It made sense to just skip the middle man and use a machine type container system to run my minikube workloads.

Why not juju?

Simply put, Juju does a fantastic job; but to stay objective I wanted to achieve minikube in LXD as a functional alternative to juju deploy kubernetes-core, or using KVM/VirtualBox in this solution.

Prereqs

You'll need to install some things to make this work. I'm going to presume you're on an Ubuntu LTS installation (16.04 plz)

sudo apt-get install -y lxd
sudo lxd init
# configure lxd with the prompts. This is mostly trivial, i did however skip ipv6 networking and opted for ipv4 only.
sudo snap install kubectl

Once you've got lxd installed and configured, you're ready to create the profile and launch your minikube "machine".

lxc profile create minikube
lxc profile edit minikube

Put the following contents in your minikube profile verbatim

name: minikube
config:
  linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: |
    lxc.aa_profile=unconfined
    lxc.mount.auto=proc:rw sys:rw
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: Profile supporting minikube in containers
devices:
  aadisable:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk

Now, launch your minikube container

lxc launch ubuntu:16.04 minikube
lxc profile apply minikube default,minikube

From here, you're ready to enter the container and setup the components

lxc exec minikube /bin/bash

Inside the container

curl https://get.docker.com | bash
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

minikube start --apiserver-name minikube --vm-driver none

This will spin up the minikube instance. If you dont get any errors in minikube log you're nearly complete!

cd /root/.minikube
kubectl config --kubeconfig=minikube set-cluster minikube --server=https://kubernetes:8443 --certificate-authority=ca.crt --embed-certs=true
kubectl config --kubeconfig=minikube unset users
kubectl config --kubeconfig=minikube set-credentials minikube --client-key=client.key --client-certificate=client.crt --embed-certs=true
kubectl config --kubeconfig=minikube set-context default --cluster=minikube --user=minikube
kubectl config --kubeconfig=minikube use-context default

Awesome! We have a portable kubeconfig now too. we're ready to exit the container

exit

Back on our host

We'll need to do 2 final things to finish the setup. We need to grab that kubeconfig from the minikube container, and we'll need to do an /etc/hosts poison to satisfy the x509 validation on the TLS certificates

To get the IP address of the container, you can re-exec into it, or run lxc list to get the IP from the listing.

+----------+---------+--------------------------------+------+------------+-----------+
|   NAME   |  STATE  |              IPV4              | IPV6 |    TYPE    | SNAPSHOTS |
+----------+---------+--------------------------------+------+------------+-----------+
| minikube | RUNNING | 172.17.0.1 (docker0)           |      | PERSISTENT | 0         |
|          |         | 10.169.52.195 (eth0)           |      |            |           |
+----------+---------+--------------------------------+------+------------+-----------+

So we'll put that in our /etc/hosts file. Included snippet for clarity if you haven't poisoned your DNS before.

127.0.0.1	localhost
127.0.1.1	bushido
10.169.52.195 kubernetes

Now grab the kubeconfig from the container that we generated and we're ready to go

lxc exec minikube cat /root/.minikube/minikube > kubeconfig
kubectl --kubeconfig kubeconfig  get no

NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     <none>    25m       v1.7.5

Viola!

You can do everything you would do with k8s in a vm (barring some testing and limitations mind you, but it should be pretty close!)

kubectl --kubeconfig kubeconfig proxy

now visit https://localhost:8001/ui in your browser and start deploying the world!

Disclaimer

This is not an official project, I'm happy to help get you started if you're interested in this or in making it an officially supported mechanism by the minikube project. However - with that being said this is the hackers warranty. You've set this up and if/when it breaks, there's no warranty and I'm not supporting this in an official project capacity.

Best of luck and happy hacking!