k8s.md

k8s.md

Kubernetes (K8s)

Installation Minikube KVM (WIP)

Installation of Minikube in KVM Centos7 VM image with --vm-driver=none

Installation KVM image

Create Centos 7 KVM Image

1. List available images

$ sudo virt-builder --list | grep -i --color centos

centos-6                 x86_64     CentOS 6.6
centos-7.0               x86_64     CentOS 7.0
centos-7.1               x86_64     CentOS 7.1
centos-7.2               aarch64    CentOS 7.2 (aarch64)
centos-7.2               x86_64     CentOS 7.2
centos-7.3               x86_64     CentOS 7.3
centos-7.4               x86_64     CentOS 7.4
centos-7.5               x86_64     CentOS 7.5
centos-7.6               x86_64     CentOS 7.6
centos-7.7               x86_64     CentOS 7.7
centos-8.0               x86_64     CentOS 8.0

Pick centos-7.7

2. Get default pool-list

$ sudo virsh pool-list | grep -i --color default
default              active     yes

3. Get default pool path

$ sudo virsh pool-dumpxml default | grep -oP "<path>(.*)</path>" | cut -d ">" -f 2 | cut -d "<" -f 1
/opt/user/kvm

4. Change directory to default pool path

$ cd /opt/user/kvm

5. Create qcow2 image

<your_password> - default password to created centos image

$ sudo virt-builder centos-7.7 --arch x86_64 --size 100G -m 8192 --root-password password:<your_password> --format qcow2

6. Create minikube KVM Centos virtual machine instance

$ sudo virt-install --name minikube --ram 8192 --vcpus=4 --disk path=/opt/user/kvm/centos-7.7.qcow2 --rng /dev/urandom --os-variant centos7.0 --import

Setup Centos 7 KVM Image environment

1. Switch off selinux for current session

Check status first:

$ setstatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

if SELinux status: enabled than:

sudo setenforce 0

2. Switch off selinux permanently

$vi /etc/selinux/config

SELINUX=disabled

3. Disable firewalld systemd service

$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld

4. Reboot VM

$ sudo reboot

Note: We switch off selinux and firewalld service, because we expect that Minikube installation will be used only for test/development purposes

Install Docker daemon

1. Find Kubernetes required Docker version Kubernetes - CHANGELOG-1.16

2. Install required packages see Docker

$ sudo yum install iptables git procps-ng xz

3. Download the Docker binaries package

$ curl -L https://download.docker.com/linux/static/stable/x86_64/docker-18.09.9.tgz -o docker-18.09.9.tgz

4. Install binaries

$ sudo tar -xvzf docker-18.09.9.tgz -C /usr/local/ && sudo ln -s /usr/local/docker/* /usr/local/bin

5. Create systemd Docker service docker.service configuration

$ sudo vi /etc/systemd/system/docker.service

# /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/local/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

6. Create systemd Docker service socket docker.socket configuration

$ sudo vi /etc/systemd/system/docker.socket

[Unit]
Description=Docker Socket for the API
PartOf=docker.service

[Socket]
# If /var/run is not implemented as a symlink to /run, you may need to
# specify ListenStream=/var/run/docker.sock instead.
ListenStream=/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

7. Add docker group

$ sudo groupadd -g 1001 docker

8. Add docker group to developer user

$ sudo usermod -a -G docker developer

9. Create containerd systemd service containerd.service configuration

sudo vi /etc/systemd/system/containerd.service

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
KillMode=process
Delegate=yes
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity

[Install]
WantedBy=multi-user.target

10. Create /run/containerd directory

$ sudo mkdir -p /run/containerd && sudo chmod -v 0711 /run/containerd

11. Start services

$ sudo systemctl daemon-reload && sudo systemctl start containerd.service && systemctl start docker.service

Installing minikube

1. yum install -y socat
2. Documentation - Getting Started - Linux - None (bare-metal)

Configuring minikube

Minikube rootless

1. Move /root/.kube to /home/developer

sudo mv /root/.kube /home/developer/.kube # this will write over any previous configuration
sudo chown -R developer:users /home/developer/.kube

2. Move /root/.kube to /home/developer

sudo mv /root/.minikube /home/developer/.minikube # this will write over any previous configuration
sudo chown -R developer:users /home/developer/.minikube

3. Modify paths in `/home/developer/.kube/config

apiVersion: v1
clusters:
- cluster:
    certificate-authority: ../.minikube/ca.crt
    server: https://192.168.122.142:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: ../.minikube/client.crt
    client-key: ../.minikube/client.key

Minikube kubectl bash completion

1. kubectl bash completion

Minikube bash completion

1. Add to ~/.bashrc

source <(minikube completion bash)

2. Relogin as current user or

source ~/.bashrc

Troubleshooting

• Failed to get system container stats for "/system.slice/docker.service"

systemctl status -l kubelet
...
Jan 04 20:51:28 localhost.localdomain kubelet[808]: E0104 14:51:28.377869     808 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.s
ervice": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
...

• Add to kubelet.service

--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

See: kubelet-fails-to-get-cgroup-stats-for-docker-and-kubelet-services

• ! VM may be unable to resolve external DNS records

yum install bind-utils

• Purge minikube generated files

see minikube failed to start on Ubuntu 18.04 with VirtualBox

Links

• Minikube
• libguestfs
• Kubernetes - Minikube
• Docker - Install binaries
• how-to-disable-selinux-on-centos-7
• Moby - systemd docker service/socket