Ipv4 address preservation in K8s KinD

README.md

Ipv4 address preservation in K8s KinD

In large cluster IPv4 addresses can become a limiting resource. In this gist we explore a way to assign IPv4 addresses to PODs on just some nodes using the MultiCIDRRangeAllocator feature. Project Calico may present other possibilities.

Configuration

KinD configuration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  kubeadmConfigPatches:
    - |
      kind: JoinConfiguration
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "wantIPv4=true"
- role: worker
networking:
  ipFamily: dual
  disableDefaultCNI: true
featureGates:
  "MultiCIDRRangeAllocator": true
runtimeConfig:
  "networking.k8s.io/v1alpha1": true
kubeadmConfigPatches:
- |
  kind: ClusterConfiguration
  metadata:
    name: config
  controllerManager:
    extraArgs:
      "cidr-allocator-type": "MultiCIDRRangeAllocator"

The MultiCIDRRangeAllocator feature-gate is enabled and the "cidr-allocator-type" is configured. Node "kind-worker" is labled "wantIPv4=true" to request IPv4 addresses.

The default CNI-plugin (kindnet) will produce an un-usable (empty) ipam config and can't be used.

Two ClusterCIDR objects are created. One that match nodes with label "wantIPv4=true" that configures both ipv4 and ipv6, and a default that configures only ipv6.

apiVersion: networking.k8s.io/v1alpha1
kind: ClusterCIDR
metadata:
  name: ipv4
spec:
  nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
        - {key: wantIPv4, operator: In, values: ["true"]}
  perNodeHostBits: 7
  ipv4: "172.20.0.0/24"
  ipv6: "fd00:1000::/120"
---
apiVersion: networking.k8s.io/v1alpha1
kind: ClusterCIDR
metadata:
  name: default
spec:
  perNodeHostBits: 16
  ipv6: "fc00:2000::/96"

Run

Start KinD with the configuration above, create the two ClusterCIDR objects and load a CNI-plugin that uses the K8s assigned ranges. I use xcluster-cni but I have got Antrea working but flannel failed.

kind create cluster --config range-allocator.yaml --image kindest/node:v1.26.0
# Define the ClusterCIDR objects.
kubectl create -f cluster-cidr.yaml
# Load a CNI-plugin
kubectl apply -n kube-system -f https://raw.githubusercontent.com/Nordix/xcluster-cni/master/xcluster-cni.yaml

Now check the address ranges assigned to the nodes:

> kubectl get node kind-worker -o json | jq .spec.podCIDRs
[
  "172.20.0.0/25",
  "fd00:1000::/121"
]
> kubectl get node kind-worker2 -o json | jq .spec.podCIDRs
[
  "fc00:2000::/112"
]

Exactly what we wanted, but...

> kubectl get pods -A -o wide
NAMESPACE            NAME                                         READY   STATUS             RESTARTS      AGE     IP             NODE                 NOMINATED NODE   READINESS GATES
kube-system          coredns-787d4945fb-t54sj                     0/1     Running            0             5m35s   fc00:2000::4   kind-worker2         <none>           <none>
kube-system          coredns-787d4945fb-wcb4t                     0/1     Running            0             5m35s   fc00:2000::3   kind-worker2         <none>           <none>
...
> kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   7m6s

PODs scheduled to "worker2" will only get an ipv6 address but the kubernetes service is ipv4. This means that those can't contact the API-server!

The way to fix this is to make ipv6 the default family. Then the kubernetes service would get an ipv6 address. I don't know how to do that in KinD (or if it can be done).