kubeadm-containerd-audit.md

Configure audit logging & troubleshooting containerd-based kubeadm

First, we'll need a VM. In one simple command, you can create a VM in azure and pass a cloud-init script that will install containerd and kubeadm, and will deploy a single node Kubernetes cluster:

wget https://gist.githubusercontent.com/ams0/0e57d15d53782c2c2259cce8545caa70/raw/d4e0686e4dc068ea146717af5d5a7be3dab97a4c/kubeadm-containerd.sh

az group create -n cks
az vm create -g cks -n cks --image  UbuntuLTS --ssh-key-values ~/.ssh/id_rsa.pub --admin-username cks --size Standard_B4ms --custom-data kubeadm-containerd.sh

SSH into the newly created VM and become root:

sudo su
$ export KUBECONFIG=/etc/kubernetes/admin.conf
$ alias k=kubectl

$ k get no -o wide
NAME   STATUS   ROLES                  AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
cks    Ready    control-plane,master   120m   v1.20.1   10.0.0.4      <none>        Ubuntu 18.04.5 LTS   5.4.0-1031-azure   containerd://1.3.3

Create an Audit Policy

We want to modify the kube-apiserver configuration to log all Metadata level events (refer to the docs for richer auditing policy example):

cat > /etc/kubernetes/audit-policy.yaml <<EOF
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF

Modify /etc/kubernetes/manifests/kube-apiserver.yaml

...
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/audit.log
    - --audit-log-maxsize=10
    - --audit-log-maxbackup=7

...

    volumeMounts:
    - mountPath: /etc/kubernetes/audit-policy.yaml
      name: audit
      readOnly: true
    - mountPath: /var/log/audit.log
      name: audit-log
      readOnly: false

...

  volumes:
  - name: audit
    hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
  - name: audit-log
    hostPath:
      path: /var/log/audit.log
      type: FileOrCreate

kubelet is the process responsible to pick up changes in static pod manifests in /etc/kuberntes/manifests; if the manifests are invalid we can look at the kubelet logs (it's a systemd unit):

journalctl -u kubelet -f

We can check if the kube-api container is running by using ctr command (as we're running containerd):

ctr -n k8s.io c list | grep api