Examples of how to test the impact of the v1.16 API deprecations

readme.md

Kubernetes v1.16 API deprecation testing

Examples of how to test the impact of the v1.16 API deprecations and ways to debug early!

If this is the first time you're hearing of these deprecations STOP and read this blog post (thanks @vllry!).

Common misunderstandings

1. The exact apiVersion: value that I use in my manifest is stored on disk (etcd).

   When you create an object in Kubernetes, the apiVersion: value that persists on disk (etcd) will be Kubernetes' preferred version, regardless of the apiVersion: value you submit.

   There is one partial exception to this, when using kubectl apply ... a last-applied: label will persist on disk (etcd) with a value of the entire manifest as applied.

2. kubectl get $OBJECT -o yaml will print my exact manifest object I created.

   When you output a manifest, the apiVersion: value will be Kubernetes' preferred version. For that reason, we can't use kubectl output to search for running deprecated objects.

3. If I upgrade my cluster with running deprecated objects, they will fail.

   During an upgrade, Kubernetes will convert all apiVersion: values to preferred versions. As a result, there is little to no impact to running applications or objects.

TL;DR: The best way to not get bit by deprecated API's is to scan and test the source manifests.

Needed before you start:

• This binary (conftest): https://github.com/instrumenta/conftest#installation

• This file of rego rules (deprek8)

  curl https://raw.githubusercontent.com/naquada/deprek8/master/policy/deprek8.rego > deprek8.rego
  
  # NOTE: If using conftest 0.16.0 (latest), you can provide the url WITHOUT creating a file
  # conftest test --update https://raw.githubusercontent.com/naquada/deprek8/master/policy/deprek8.rego -
  

Check a running cluster for deprecated APIs

IMPORTANT: Do not use -o yaml for checking, as the output it generates will use kubectl's preferred API.

# NOTE: THIS WILL ONLY FIND OBJECTS THAT WERE CREATED USING kubectl apply (NOT create)!

# check all namespace scoped resources
for namespace in $(kubectl get ns -o=jsonpath='{.items[*].metadata.name}'); do
  kubectl apply view-last-applied networkpolicy,ds,deployment,statefulset,rs,ing -n $namespace | conftest test -p deprek8.rego -;
done

# manually review your PSP source manifests

# you can always grep some output for the lack of a preferred API
# <some source yaml> | grep -E -v 'apps/v1|networking.k8s.io/v1|policy/v1beta1|networking.k8s.io/v1beta1'

Configure kubeadm to pre-deprecate the API's in earlier versions

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
  name: config
apiServer:
  extraArgs:
    runtime-config: "apps/v1beta1=false,apps/v1beta2=false,extensions/v1beta1/daemonsets=false,extensions/v1beta1/deployments=false,extensions/v1beta1/replicasets=false,extensions/v1beta1/networkpolicies=false,extensions/v1beta1/podsecuritypolicies=false"

# kubeadm init <...> --config <ClusterConfig>.yaml

# validate with:
# kubectl exec -it <APISERVER-NAME> -n kube-system | ps -ef | grep runtime-config

Testing valid and invalid APIs

Good (should pass)

# generic apps/v1 deployment
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - image: devnull
        name: devnull
EOF

# Now with more with rego!
cat <<EOF | conftest test -p deprek8.rego -
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - image: devnull
        name: devnull
EOF

Bad (should fail)

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - image: devnull
        name: devnull
EOF

# Now with more with rego!
cat <<EOF | conftest test -p deprek8.rego -
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - image: devnull
        name: devnull
EOF

Using kubectl to fix a "bad" deployment with convert

cat <<EOF | kubectl convert -f - | kubectl apply -f -
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - image: devnull
        name: devnull
EOF

But wait! convert is being deprecated! Is there another way?

# Use a period to append a specific object spec

kubectl get deployments test -o yaml
    ...
    apiVersion: extensions/v1beta1
    kind: Deployment
    ...

kubectl get deployments.apps test -o yaml
    ...
    apiVersion: apps/v1
    kind: Deployment
    ...

Test with KinD

If you haven't heard about KinD, start here: https://kind.sigs.k8s.io/

cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  metadata:
    name: config
  apiServer:
    extraArgs:
      runtime-config: "apps/v1beta1=false,apps/v1beta2=false,extensions/v1beta1/daemonsets=false,extensions/v1beta1/deployments=false,extensions/v1beta1/replicasets=false,extensions/v1beta1/networkpolicies=false,extensions/v1beta1/podsecuritypolicies=false"
EOF

# using v1.15.6 since the APIs are gone in v1.16.0
kind create cluster --image=kindest/node:v1.15.6@sha256:18c4ab6b61c991c249d29df778e651f443ac4bcd4e6bdd37e0c83c0d33eaae78 --config kind-config.yaml