SGX EPC cgroups for Kubernetes

README.txt

1. Prepare the kernel

git clone --depth 1 -b sgx_cg_upstream_v12 https://github.com/haitaohuang/linux.git linux-epc-cgroups

Added config:
CONFIG_CGROUP_SGX_EPC=y

2. Boot the VM and check SGX cgroups

host:$ qemu-system-x86_64 \
    ...
    -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
    -M sgx-epc.0.memdev=mem1 \
    -drive file=jammy.raw,if=virtio,aio=threads,format=raw,index=0,media=disk \
    -kernel ./arch/x86_64/boot/bzImage \
    ...

guest:$ grep sgx_epc /sys/fs/cgroup/misc.capacity
sgx_epc 67108864  

3. Setup (a single node) K8S cluster w/ containerd 1.7 and SGX EPC NRI plugin on Ubuntu 22.04

$ dpkg -l |grep containerd
ii  containerd                      1.7.2-0ubuntu1~22.04.1                  amd64        daemon to control runC
# NB: config.toml: enable nri (disable = false), systemdCgroup = true
$ grep -A7 nri\.v1 /etc/containerd/config.toml
  [plugins."io.containerd.nri.v1.nri"]
    disable = false
    disable_connections = false
    plugin_config_path = "/etc/nri/conf.d"
    plugin_path = "/opt/nri/plugins"
    plugin_registration_timeout = "5s"
    plugin_request_timeout = "2s"
    socket_path = "/var/run/nri/nri.sock"
$ sudo ls /var/run/nri/
nri.sock

$ git clone -b PR-2023-050 https://github.com/mythi/intel-device-plugins-for-kubernetes.git
$ cd intel-device-plugins-for-kubernetes
$ make intel-deviceplugin-operator
$ docker save intel/intel-deviceplugin-operator:devel > op.tar
$ sudo ctr -n k8s.io i import op.tar
$ kubectl apply -k deployments/operator/default/
$ kubectl apply -f deployments/operator/samples/deviceplugin_v1_sgxdeviceplugin.yaml

4. Run

Use https://raw.githubusercontent.com/containers/nri-plugins/main/scripts/testing/kube-cgroups and run
watch -n 1 "./kube-cgroups -n 'sgxplugin-*' -f '(misc|memory).(max|current)'" -p 'sgx-epc-*'
(with the targeted namespace (-n) and podname filter (-p))

Run a pod requesting sgx.intel.com/epc: "65536"

5. e2e test framework

$ git clone -b PR-2023-050 https://github.com/mythi/intel-device-plugins-for-kubernetes.git
$ cd intel-device-plugins-for-kubernetes
$ make stress-ng-gramine intel-sgx-admissionwebhook
$ docker save intel/intel-sgx-admissionwebhook:devel > wh.tar
$ sudo ctr -n k8s.io i import wh.tar
$ docker save intel/stress-ng-gramine:devel > gr.tar
$ sudo ctr -n k8s.io i import gr.tar
$ go test -v ./test/e2e/... -ginkgo.v -ginkgo.focus "Device:sgx.*App:sgx-epc-cgroup"

NB: The e2e test framework expects cert-manager is deployed in the cluster
NB: The e2e test framework deletes all but kube-system and cert-manager namespaces before running the tests so do not run in a cluster with something important deployed!

@mythi I sent you my notes that I wrote as I went through the steps. This guide is great. Some improvements I can think of is including notes for installing cert-manager and NFD -- I did not know this as I had never used intel-device-plugin before this. Thanks!