egernst

After failing to get e2e to work with a preexisting cluster, ala kubetest, I learned that this isn't the method most folks use, despite what is all over the kubernetes documentation. I was pushed towards e2e-k8s.sh from the KIND repo.

To get started,

go get -u github.com/kubernetes-sigs/kind go get k8s.io/kubernetes

KIND can start a cluster from sources, assuming you are calling kind from the root of your k/k tree. ur k/k tree

egernst

Accessing Kubelet API

curl -sk https://localhost:10250/pods/

• If --anonymous-auth is turned off, you will see a 401 Unauthorized response.
• If --anonymous-auth is true and --authorization-mode is Webhook you'll see 403 Forbidden response with message Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy)
• If --anonymous-auth is true and --authorization-mode is AlwaysAllow you'll see a list of pods.

egernst

klet.resourceAnalyzer = serverstats.NewResourceAnalyzer(klet, kubeCfg.VolumeStatsAggPeriod.Duration)

// NewResourceAnalyzer returns a new ResourceAnalyzer func NewResourceAnalyzer(statsProvider Provider, calVolumeFrequency time.Duration) ResourceAnalyzer { fsAnalyzer := newFsResourceAnalyzer(statsProvider, calVolumeFrequency) summaryProvider := NewSummaryProvider(statsProvider) return &resourceAnalyzer{fsAnalyzer, summaryProvider} }

egernst

$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/busybox-two" | jq ' '

{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "busybox-two",

egernst

#!/bin/bash

set -e

export ghprbPullId
export ghprbTargetBranch
export KATA_DEV_MODE="false"
export KATA_HYPERVISOR="firecracker"
export CI="true"
export CI_JOB="FIRECRACKER"

egernst

As root:

for c in {0..87}; do echo performance > /sys/devices/system/cpu/cpu$c/cpufreq/scaling_governor; done

Verify:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

egernst

kubeadm-config.yaml:

---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# Allowing for CPU pinning and isolation in case of guaranteed QoS class
-featureGates:
-  PodOverhead: true
cpuManagerPolicy: static
systemReserved:

egernst

Eviction handling

Kubelet manages eviction, which is carried out at pod-granularity on a node. The kubelet ranks Pods for eviction first by whether or not their usage of the starved resource exceeds requests, then by Priority, and then by the consumption of the starved compute resource relative to the Pods’ scheduling requests.

Of note for Pod Overhead is the comparision of requested resources versus utilization of particular resource. The sum of requests is compared against the sum of container utilization, for each pod.

Eviction is handled by an Evicition Manager.

NewManager is passed a summaryProvider, which is a part of the StatsProvider created for Kubelet. In our case, it should be a New CRI Stats Provider (see ~/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/summary.go for analyzer?)

egernst

sudo -E apt install -y curl
sudo bash -c "cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main
EOF"
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo -E apt update
sudo -E apt install -y kubelet kubeadm kubectl

VERSION="1.2.7"

egernst

Quick guide for getting Kata+containerd (using v2 shim) up and running super quick on bionic

Installation of Kube stuff on bionic:

Use the following sh:

sudo -E apt install -y curl
sudo bash -c "cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main