turingpi.md

Turing Pi Kubernetes Cluster

Board

• Turing Pi 2
• Pico PSU 160w

Modules

• 3x Turing RK1 8 cores / 32 GB
• 1x N-Vidia Jetson Orin NX 8 cores / 16 GB / 1024 CUDA cores / 32 Tensor cores
• 4x Samsung 970 EVO Plus NVMe M.2 SSD 250GB

32 cores / 112 GB / 1024 CUDA cores / 32 Tensor cores

1TB storage

Case

• Densium APU 4L
• Noctua NF-A14 ULN 140mm

Still waiting on the RK1s, so can't really install much of a Kubernetes cluster... but the current plan is

• k0s
• cilium

k0s

This is kinda great...

Config

export NODE1=10.0.0.60
export NODE2=10.0.0.61
export NODE3=10.0.0.62
export NODE4=10.0.0.63

# k0s.yaml

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: turingpicluster
spec:
  # https://github.com/k0sproject/k0sctl#host-fields
  hosts:
    - ssh:
        address: $NODE1
        user: k0s
        port: 22
        keyPath: ~/.ssh/id_rsa
      role: controller+worker
      noTaints: true
      installFlags:
        - --cri-socket remote:unix:///var/run/crio/crio.sock
        - --profile crio-compatibility
    - ssh:
        address: $NODE2
        user: k0s
        port: 22
        keyPath: ~/.ssh/id_rsa
      role: controller+worker
      noTaints: true
      installFlags:
        - --cri-socket remote:unix:///var/run/crio/crio.sock
        - --profile crio-compatibility
    - ssh:
        address: $NODE3
        user: k0s
        port: 22
        keyPath: ~/.ssh/id_rsa
      role: controller+worker
      noTaints: true
      installFlags:
        - --cri-socket remote:unix:///var/run/crio/crio.sock
        - --profile crio-compatibility
    - ssh:
        address: $NODE4
        user: k0s
        port: 22
        keyPath: ~/.ssh/id_rsa
      role: worker
      installFlags:
        - --cri-socket remote:unix:///var/run/crio/crio.sock
        - --profile crio-compatibility
  k0s:
    version: 1.27.4+k0s.0
    dynamicConfig: false
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s
      spec:
        api:
          k0sApiPort: 9443
          port: 6443
        installConfig:
          users:
            etcdUser: etcd
            kineUser: kube-apiserver
            konnectivityUser: konnectivity-server
            kubeAPIserverUser: kube-apiserver
            kubeSchedulerUser: kube-scheduler
        konnectivity:
          adminPort: 8133
          agentPort: 8132
        network:
          kubeProxy:
            disabled: true
          podCIDR: 10.244.0.0/16
          provider: custom
          serviceCIDR: 10.96.0.0/12
        podSecurityPolicy:
          defaultPolicy: 00-k0s-privileged
        storage:
          type: etcd
        telemetry:
          enabled: false
        workerProfiles:
          - name: crio-compatibility
            values:
              cgroupDriver: systemd
        extensions:
          helm:
            concurrencyLevel: 5
            repositories:
              - name: stable
                url: https://charts.helm.sh/stable
              - name: cilium
                url: https://helm.cilium.io
            charts:
              - name: cilium
                chartname: cilium/cilium
                namespace: kube-system
                version: "1.14.0"
                values: |
                  kubeProxyReplacement: true
                  k8sServiceHost: $NODE1
                  k8sServicePort: "6443"
                  loadBalancer:
                    mode: dsr
                  containerRuntime:
                    integration: crio
                  hubble:
                    enabled: true
                    relay:
                      enabled: true
                    ui:
                      enabled: true

Install

k0sctl apply --config ./k0s.yaml
k0sctl kubeconfig --config ./k0s.yaml

Nvidia

The version of the nvidia toolkit needs to be updated... forcefully.

Install

https://nvidia.github.io/nvidia-container-runtime/

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/experimental/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update

apt -y install python3-pip curl
sudo pip3 install -U jetson-stats

Helpful commands

jtop

kube-router

Jetson ubuntu does not have ipset kernel module installed.

Installation

sudo cp /etc/apt/sources.list /etc/apt/sources.list~
sudo sed -Ei 's/^# deb-src /deb-src /' /etc/apt/sources.list
sudo apt update
sudo apt build-dep ipset

Edit /etc/depmod.d/ubuntu.conf and add ‘extra’ to the end of the line, so it looks like:

search updates ubuntu built-in extra

This part needs to be ran after every kernel upgrade

Load dependencies

sudo depmod

Build ipset kernel module and install

sudo apt source ipset
cd ipset-7.*
sudo ./autogen.sh
sudo ./configure
sudo make modules
sudo make modules_install

modprobe xt_set
lsmod

CNI

If ever changing CNIs... clear out this dir

sudo rm -rf /etc/cni/net.d