Run Kubernets conformance tests

README.md

How to run Kubernetes conformance test

Conformance test are a subset of the e2e test Kubernetes test.

The e2e tests are based on Ginkgo https://onsi.github.io/ginkgo/ and live in the Kubernetes repository https://github.com/kubernetes/kubernetes/tree/master/test/e2e.

There is no guarantee of compatibility for the e2e binaries, hence the e2e binary version MUST match your kubernetes cluster version.

To run Conformance tests against your cluster you just need to pass the Kubernetes version and the path to the kubeconfig for your cluster.

./conformance.sh v1.21.1 kubeconfig.config
+ '[' 2 -ne 2 ']'                                                                                                                                                           
+ KUBERNETES_VERSION=v1.21.1                                                                                                                                                
++ readlink -f ./kconfig                                                                                                                                                    
+ KUBECONFIG_PATH=/home/aojea/Downloads/kconfig                                                                                                                             
+ TMP_DIR=/tmp/conformancev1.21.1                                                                                                                                           
+ '[' '!' -d /tmp/conformancev1.21.1 ']'                                                                                                                                    
+ mkdir -p /tmp/conformancev1.21.1                                                                                                                                          
+ curl -L https://dl.k8s.io/v1.21.1/kubernetes-test-linux-amd64.tar.gz -o /tmp/conformancev1.21.1/kubernetes-test-linux-amd64.tar.gz                                        
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                             
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                               
100   154  100   154    0     0    534      0 --:--:-- --:--:-- --:--:--   532                                                                                              
100  242M  100  242M    0     0  11.4M      0  0:00:21  0:00:21 --:--:-- 11.9M                                                                                              
+ tar xvzf /tmp/conformancev1.21.1/kubernetes-test-linux-amd64.tar.gz --directory /tmp/conformancev1.21.1 --strip-components=3 kubernetes/test/bin/ginkgo kubernetes/test/bi
n/e2e.test                                                                                                                                                                  
kubernetes/test/bin/ginkgo                                                                                                                                                  
kubernetes/test/bin/e2e.test                                                                                                                                                
+ export KUBERNETES_CONFORMANCE_TEST=y                                                                                                                                      
+ KUBERNETES_CONFORMANCE_TEST=y                                                                                                                                             
+ /tmp/conformancev1.21.1/e2e.test -ginkgo.v -ginkgo.focus '[Conformance]' -ginkgo.skip= -provider skeleton -kubeconfig /home/aojea/Downloads/kconfig                       
I0630 10:07:49.875789 3066457 e2e.go:129] Starting e2e run "51cbe07e-f2f1-4328-9a86-0c56e45cba49" on Ginkgo node 1                                                          
{"msg":"Test Suite starting","total":5771,"completed":0,"skipped":0,"failed":0}                                                                                             
Running Suite: Kubernetes e2e suite                                                                                                                                         
===================================                                                                                                                                         
Random Seed: 1625040468 - Will randomize all specs                                                                                                                          
Will run 5771 of 5771 specs                                                                                                                                                 
                                                                                                                                                                            
Jun 30 10:07:49.887: INFO: >>> kubeConfig: /home/aojea/Downloads/kconfig                                                                                                    
Jun 30 10:07:49.889: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable                                                                                      
Jun 30 10:07:49.900: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready                                            
Jun 30 10:07:49.918: INFO: 6 / 6 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)                                                                  
Jun 30 10:07:49.918: INFO: expected 2 pod replicas in namespace 'kube-system', 2 are Running and Ready.                                                                     
Jun 30 10:07:49.918: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
Jun 30 10:07:49.922: INFO: e2e test version: v1.21.1
Jun 30 10:07:49.923: INFO: kube-apiserver version: v1.22.0-alpha.2.452+9d27400fe20867
Jun 30 10:07:49.923: INFO: >>> kubeConfig: /home/aojea/Downloads/kconfig
Jun 30 10:07:49.925: INFO: Cluster IP family: ipv4

Advance usage

The e2e binary used for the Conformance test can run more e2e test, you just need to play with the ginkgo SKIP and FOCUS parameters.

You can also run the tests in parallel.

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#running-conformance-tests

conformance.sh

#!/bin/bash

set -o errexit -o nounset -o xtrace

if [ $# -ne 2 ]; then
  "Usage: $0 [kubernetes_version] [kubeconfig_path]"
  exit 1
fi


KUBERNETES_VERSION="$1"
KUBECONFIG_PATH=$(readlink -f "$2")
FOCUS=\[Conformance\]
SKIP=\[Serial\]
NUM_NODES=20

TMP_DIR=/tmp/conformance"$1"
E2E_REPORT_DIR=${E2E_REPORT_DIR:-$TMP_DIR}

if [ ! -d $TMP_DIR ]; then
  mkdir -p $TMP_DIR
  # Download e2e test binary
  curl -L https://dl.k8s.io/${KUBERNETES_VERSION}/kubernetes-test-linux-amd64.tar.gz -o $TMP_DIR/kubernetes-test-linux-amd64.tar.gz
  tar xvzf $TMP_DIR/kubernetes-test-linux-amd64.tar.gz \
          --directory $TMP_DIR \
          --strip-components=3 kubernetes/test/bin/ginkgo kubernetes/test/bin/e2e.test
fi

export KUBERNETES_CONFORMANCE_TEST='y'

# $TMP_DIR/e2e.test -ginkgo.v \
#        -ginkgo.focus ${FOCUS:-\[Conformance\]} \
#        -ginkgo.skip="${SKIP:-}" \
#        -provider skeleton \
#        -kubeconfig ${KUBECONFIG_PATH} \
#        ${CONTAINER_RUNTIME:+"--container-runtime=${CONTAINER_RUNTIME}"} \
#        ${NUM_NODES:+"--num-nodes=${NUM_NODES}"} \
#        ${E2E_REPORT_DIR:+"--report-dir=${E2E_REPORT_DIR}"} \
#        ${E2E_REPORT_PREFIX:+"--report-prefix=${E2E_REPORT_PREFIX}"}

# to run test in parallel we have to use ginkgo and skip \[Serial\] tests
$TMP_DIR/ginkgo --nodes=${NUM_NODES}   \
  --focus=${FOCUS}                     \
  --skip=${SKIP}                       \
  $TMP_DIR/e2e.test                    \
  --                                   \
  --kubeconfig=${KUBECONFIG}           \
  --provider=local                     \
  --dump-logs-on-failure=false         \
  --report-dir=${E2E_REPORT_DIR}       \
  --disable-log-dump=true