Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Installing microk8s in an LXC container

Installing microk8s in an LXC container

I wanted to run Microk8s on a Proxmox 6 host inside of an LXC container. These are my notes from the journey.

  1. Create a privileged LXC container through the Proxmox web interface
  • Enable nesting and FUSE
    • In Proxmox UI, select container, then Options > Features > Check nesting and FUSE boxes
  1. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/.conf
    • Add the following lines
      • lxc.apparmor.profile: unconfined
      • lxc.cap.drop:
      • lxc.mount.auto: proc:rw sys:rw
  2. Start (or restart) the container
  3. SSH into the container and create a symlink for /dev/kmsg, which is missing in Ubuntu 19.10 containers
    • ln -s /dev/console /dev/kmsg
    • Has to be repeated on container reboot, which is annoying.
  4. Install snapd: apt install snapd
  5. Install microk8s: snap install microk8s --classic

The snap commands may need to be run more than once to get past errors.

Troubleshooting

If you get "cannot change profile for the next exec call: No such file or directory", try running: apparmor_parser -r /var/lib/snapd/apparmor/profiles/*

@jez500
Copy link

jez500 commented Oct 24, 2020

@acj
Thank you! I was stuck on this for hours.
Did you find a solution to the symlink on reboot? I was just going to add it to /etc/rc.local

@acj
Copy link
Author

acj commented Oct 30, 2020

@jez500 I'm glad that it was helpful! I haven't done anything to fix the symlink issue yet, but let me know if the rc.local solution works well for you. My plan was to create something with systemd or cron on the host that would periodically fix the container, which doesn't seem ideal.

@jez500
Copy link

jez500 commented Oct 30, 2020

@acj rc.local didn't seem to work, server rebooted over the last week and started getting the "cannot change profile" errors with the /dev/kmsg symlink missing. Again your notes helped me out :) I created this script which I might hook up to a cron in the future but for now it seems to do the trick

#!/bin/bash

# If this symlink is missing, likely microk8s isn't running
if [ ! -L /dev/kmsg ]; then
  ln -s /dev/console /dev/kmsg
  apparmor_parser -r /var/lib/snapd/apparmor/profiles/*
  microk8s stop
  microk8s start
fi

@VV0JC13CH
Copy link

VV0JC13CH commented Nov 15, 2021

In Proxmox >7.0 (LXC container with Ubuntu 20.04.3 LTS) I had issue with below warning:
WARNING: The memory cgroup is not enabled. The cluster may not be functioning properly. Please ensure cgroups are enabled See for example: https://microk8s.io/docs/install-alternatives#heading--arm
In order to fix my issue I had to switch back from cgroupv2 to legacy solution.
Open shell of proxmox host and change GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub to below one
GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0 quiet"
then update-grub && reboot.
Docs:
https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline

@simonjcarr
Copy link

simonjcarr commented Dec 5, 2021

I found that if you put a @reboot line in your crontab it will add the ln when the container boots.

@reboot ln -s /dev/console /dev/kmsg

To add the line above, type crontab -e which will open your crontab in an editor, then add the line above to the bottom of the file.

@simonjcarr
Copy link

simonjcarr commented Dec 5, 2021

@VV0JC13CH Your suggestion to edit grub helped in getting rid of the cgroup error, however I am still having problems.

When I run microk8s start I get the message Started., the terminal freezes for about a minute and then I get my prompt back.

When I run microk8s status I get

microk8s is not running. Use microk8s inspect for a deeper inspection.

I run microk8s inspect and get

microk8s is not running. Use microk8s inspect for a deeper inspection.
root@mk8s-3-1:~# microk8s inspect
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-kubelite is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow

Building the report tarball
  Report tarball is at /var/snap/microk8s/2695/inspection-report-20211205_114227.tar.gz

microk8s kubectl get nodes I get

The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

If I run it again I get

NAME       STATUS     ROLES    AGE   VERSION
mk8s-3-1   NotReady   <none>   31m   v1.22.4-3+adc4115d990346

and then it switches between these two states, so looks like microk8s is trying to start, failing and then restarting

Can anyone think of anything I can try to get this working

@88plug
Copy link

88plug commented Jun 2, 2022

A few updates to the original steps using Debian 11 LXC Template / June 2022

  1. Create a privileged LXC container through the Proxmox web interface, for swap amount enter 0.
  • Enable nesting and FUSE
    • In Proxmox UI, select container, then Options > Features > Check nesting and FUSE boxes
  1. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/.conf
    • Add the following lines
lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file 0 0
  1. Start (or restart) the container
  2. Install snapd: apt install -y snapd squashfuse fuse / reboot!
  3. Install microk8s: snap install microk8s --classic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment