Skip to content

Instantly share code, notes, and snippets.

@sys0dm1n

sys0dm1n/kubernetes-smoke-testing.md

Last active May 2, 2019 11:46
Show Gist options
  • Save sys0dm1n/79157026f73d44fbe82b289664b849db to your computer and use it in GitHub Desktop.
Save sys0dm1n/79157026f73d44fbe82b289664b849db to your computer and use it in GitHub Desktop.
Download ZIP
Raw
kubernetes-smoke-testing.md

Smoke testing data encryption

Verify the cluster's ability to perform data encryption. Create a test secret:

kubectl create secret generic kubernetes-the-hard-way --from-literal="mykey=mydata"

Log in to one of your controller servers, and get the raw data for the test secret from etcd:

sudo ETCDCTL_API=3 etcdctl get \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.pem \
  --cert=/etc/etcd/kubernetes.pem \
  --key=/etc/etcd/kubernetes-key.pem\
  /registry/secrets/default/kubernetes-the-hard-way | hexdump -C

Your output should look something like this:

00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 6b 75 62 65 72 6e  |s/default/kubern|
00000020  65 74 65 73 2d 74 68 65  2d 68 61 72 64 2d 77 61  |etes-the-hard-wa|
00000030  79 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |y.k8s:enc:aescbc|
00000040  3a 76 31 3a 6b 65 79 31  3a fc 21 ee dc e5 84 8a  |:v1:key1:.!.....|
00000050  53 8e fd a9 72 a8 75 25  65 30 55 0e 72 43 1f 20  |S...r.u%e0U.rC. |
00000060  9f 07 15 4f 69 8a 79 a4  70 62 e9 ab f9 14 93 2e  |...Oi.y.pb......|
00000070  e5 59 3f ab a7 b2 d8 d6  05 84 84 aa c3 6f 8d 5c  |.Y?..........o.\|
00000080  09 7a 2f 82 81 b5 d5 ec  ba c7 23 34 46 d9 43 02  |.z/.......#4F.C.|
00000090  88 93 57 26 66 da 4e 8e  5c 24 44 6e 3e ec 9c 8e  |..W&f.N.\$Dn>...|
000000a0  83 ff 40 9a fb 94 07 3c  08 52 0e 77 50 81 c9 d0  |..@....<.R.wP...|
000000b0  b7 30 68 ba b1 b3 26 eb  b1 9f 3f f1 d7 76 86 09  |.0h...&...?..v..|
000000c0  d8 14 02 12 09 30 b0 60  b2 ad dc bb cf f5 77 e0  |.....0.`......w.|
000000d0  4f 0b 1f 74 79 c1 e7 20  1d 32 b2 68 01 19 93 fc  |O..ty.. .2.h....|
000000e0  f5 c8 8b 0b 16 7b 4f c2  6a 0a                    |.....{O.j.|
000000ea

Look for k8s:enc:aescbc:v1:key1 on the right of the output to verify that the data is stored in an encrypted format!

Smoke testing deployment

Verify that deployments work. Create a a simple nginx deployment:

kubectl run nginx --image=nginx

Verify that the deployment created a pod and that the pod is running:

kubectl get pods -l run=nginx

Verify that the output looks something like this:

NAME                     READY     STATUS    RESTARTS   AGE
nginx-65899c769f-9xnqm   1/1       Running   0          30s

The pod should have a STATUS of Running with 1/1 containers READY.

Smoke testing port forwarding

Verify that remote access works via port forwarding. First, get the pod name of the nginx pod and store it an an environment variable:

POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath="{.items[0].metadata.name}")

Forward port 8081 to the nginx pod:

kubectl port-forward $POD_NAME 8081:80

Open up a new terminal on the same machine running the kubectl port-forward command and verify that the port forward works.

curl --head http://127.0.0.1:8081

You should get an http 200 OK response from the nginx pod. When you are done, you can stop the port-forward in the original terminal with control+c.

Smoke testing logs

Verify that you can access container logs with kubectl logs. First, let's set an environment variable to the name of the nginx pod:

POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath="{.items[0].metadata.name}")

Get the logs from the nginx pod:

kubectl logs $POD_NAME

This command should return the nginx pod's logs. It will look something like this (but there could be more lines):

127.0.0.1 - - [10/Sep/2018:19:29:01 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"

Smoke testing exec

Verify that you can execute commands inside a container with kubectl exec. First, let's set an environment variable to the name of the nginx pod:

POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath="{.items[0].metadata.name}")

To test kubectl exec, execute a simple nginx -v command inside the nginx pod:

kubectl exec -ti $POD_NAME -- nginx -v

This command should return the nginx version output, which should look like this:

nginx version: nginx/1.15.3

Smoke testing services

Verify that services work. First, create a service to expose the nginx deployment:

kubectl expose deployment nginx --port 80 --type NodePort

Get the node port assigned to the newly-created service and assign it to an environment variable:

kubectl get svc

The output should look something like this:

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP        20d
nginx        NodePort    10.32.0.81   <none>        80:32642/TCP   2m

Look for the service called nginx in that output. Under PORT(S), look for the second port, listed after 80:. In the example above, it is 32642. That is the node port, so make note of that value since you will need it in a moment.

Next, log in to one of your worker servers and make a request to the service using the node port. Be sure to replace the placeholder with the actual node port:

curl -I localhost:<node port>

You should get an http 200 OK response.

Smoke testing untrusted workloads

Verify that untrusted workloads run using gVisor. Our Kubernetes cluster has been configured to run untrusted workloads under a more secure configuration using runsc. In this lesson, we will make sure that this functionality is working correctly. We will create a pod that is marked as untrusted, then we will log in to the worker node and dig into the runsc data to make sure that the container is actually running under runsc. This will verify that our cluster is running untrusted workloads with the correct configuration on the worker node. First, create an untrusted pod:

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
    - name: webserver
      image: gcr.io/hightowerlabs/helloworld:2.0.0
EOF

Make sure that the untrusted pod is running:

kubectl get pods untrusted -o wide

Take note of which worker node the untrusted pod is running on, then log into that worker node.

On the worker node, list all of the containers running under gVisor:

sudo runsc --root  /run/containerd/runsc/k8s.io list

Get the pod ID of the untrusted pod and store it in an environment variable:

POD_ID=$(sudo crictl -r unix:///var/run/containerd/containerd.sock \
  pods --name untrusted -q)

Get the container ID of the container running in the untrusted pod and store it in an environment variable:

CONTAINER_ID=$(sudo crictl -r unix:///var/run/containerd/containerd.sock \
  ps -p ${POD_ID} -q)

Get information about the process running in the container:

sudo runsc --root /run/containerd/runsc/k8s.io ps ${CONTAINER_ID}

Since we were able to get the process info using runsc, we know that the untrusted container is running securely as expected.

Smoke testing cleanup

You can clean up the objects that were created for smoke testing purposes like so:

kubectl delete secret kubernetes-the-hard-way
kubectl delete svc nginx
kubectl delete deployment nginx
kubectl delete pod untrusted

Verify the deployment with NetChecker

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/netcheck.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment