Kubernetes API Server Test-Endpoint - minimal dev k8s controlplane

README.md

Minimal Kubernetes Controlplane

When you create a operator, interact with the k8s api via your service or want to tinker with the controlplane server args you might want a sandboxed endpont that dosen't cost a fortune and doesen't bring down production.
But starting the controlplane isn't as easy as starting a http web server. It has dependencies to etcd end requires communication via TLS certs. Setting this up can be a pain in the a**.

Thats why I created this docker compose file and some supporting scripts to create the required certs.

How

Download the files into a dircetory that looks like ths:

root/
├─ certs/
│  ├─ generate-certs.sh
│  ├─ csr.conf
├─ docker-compose.yaml

Enter the certs directory. Make any changes to the csr.conf if needed (like IPs or hostnames) and run the script.
NOTE: Run the script under Linux, not Windows since Windows openSSL is buggy.

Now you can run docker compse up

Usecases

I used it to test OIDC integration with K8s but you can do basicly anything. But you have to keep in mind that dis is a controlplane that is basicly incapable of doing anything. It can not scedule or start pods! You can use this to:

• Test CRDs
• Test interacting with a real k8s api
• Test OIDC
• Test user auth roles
• Test manifest schemas

There are alternative local k8s tools like kind, minikube and k0s but sometimes they manage/abstract the controlplane to much. It can become hard alter data for the controllplane. Some of these tools are kind of heavy on resources since they also deploy the scheduler, kube-proxy and csi provisioners. This only uses the api server and etcd.

csr.conf

[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C = DE
ST = BE
L = BE
O = DEV
OU = DEV
CN = kube-apiserver

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.5 = kube-apiserver
DNS.6 = kube-apiserver.localhost
IP.1 = 172.19.0.1
IP.2 = 172.19.0.2
IP.3 = 172.19.0.3
IP.4 = 172.19.0.4
IP.5 = 172.19.0.5
IP.6 = 172.19.0.6
IP.7 = 172.19.0.7
IP.8 = 172.19.0.8
IP.9 = 172.19.0.9
IP.10 = 172.19.0.10
IP.11 = 10.0.0.1
IP.12 = 10.0.0.2
IP.13 = 10.0.0.3
IP.14 = 10.0.0.4
IP.15 = 127.0.0.1

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names

docker-compose.yaml

version: "3.7"

services:
  etcd:
    image: registry.k8s.io/etcd:3.5.10-0
    volumes:
      - etcd-data:/etcd-data
    command:
      - /usr/local/bin/etcd
      - --data-dir
      - /etcd-data
      - --name
      - my-etcd
      - --advertise-client-urls
      - http://etcd:2379
      - --listen-client-urls
      - http://0.0.0.0:2379
    networks:
      - k8s-net

  kube-apiserver:
    image: registry.k8s.io/kube-apiserver:v1.29.1
    command:
      - kube-apiserver
      - --etcd-servers=http://etcd:2379
      - --service-cluster-ip-range=10.0.0.0/24
      - --allow-privileged=true
      - --service-account-key-file=/certs/service-account-key.pem
      - --service-account-signing-key-file=/certs/service-account-key.pem
      - --service-account-issuer=api
      - --tls-private-key-file=/certs/server.key
      - --tls-cert-file=/certs/server.crt
      - --client-ca-file=/certs/ca.crt
      - --bind-address=0.0.0.0
      - --secure-port=8443
      # - --authorization-mode=Node,RBAC
    volumes:
      - ./certs:/certs
    ports:
      - "8443:8443"
    depends_on:
      - etcd
    networks:
      - k8s-net

networks:
  k8s-net:
    driver: bridge

volumes:
  etcd-data:

generate-certs.sh

#!/bin/bash

# service account
DATA_DIR=certs/

mkdir -p $DATA_DIR
openssl genrsa -out ${DATA_DIR}service-account-key.pem 4096
openssl req -new -x509 -days 365 -key ${DATA_DIR}service-account-key.pem -subj "/CN=test" -sha256 -out ${DATA_DIR}service-account.pem

# api-server
openssl genrsa -out ${DATA_DIR}ca.key 2048
openssl req -x509 -new -nodes -key ${DATA_DIR}ca.key -subj "/CN=test" -days 10000 -out ${DATA_DIR}ca.crt
openssl genrsa -out ${DATA_DIR}server.key 2048
openssl req -new -key ${DATA_DIR}server.key -out ${DATA_DIR}server.csr -config csr.conf
openssl x509 -req -in ${DATA_DIR}server.csr -CA ${DATA_DIR}ca.crt -CAkey ${DATA_DIR}ca.key \
    -CAcreateserial -out ${DATA_DIR}server.crt -days 10000 \
    -extensions v3_ext -extfile csr.conf

# kubeconfig
kubectl config set-cluster local-apiserver \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://127.0.0.1:8443 \
    --kubeconfig=${DATA_DIR}kubeconfig

kubectl config set-credentials admin \
    --client-certificate=server.crt \
    --client-key=server.key \
    --embed-certs=true \
    --kubeconfig=${DATA_DIR}kubeconfig

kubectl config set-context default \
    --cluster=local-apiserver \
    --user=admin \
    --kubeconfig=${DATA_DIR}kubeconfig

kubectl config use-context default --kubeconfig=${DATA_DIR}kubeconfig