Skip to content

Instantly share code, notes, and snippets.

@feiskyer
Last active Apr 1, 2016
Embed
What would you like to do?
E2e test for Kubernetes master with hyper runtime.

Install hyper

# Install hyper on CentOS 7.2
yum install -y wget libvirt
wget https://s3.amazonaws.com/hypernetes/hyper-0.5-1.el7.centos.x86_64.rpm
wget https://s3.amazonaws.com/hypernetes/hyperstart-0.5-1.el7.centos.x86_64.rpm
wget https://s3.amazonaws.com/hypernetes/qemu-hyper-2.4.1-2.el7.centos.x86_64.rpm
rpm -ivh *.rpm

cat >/etc/libvirt/qemu.conf <<EOF
user = "root"
group = "root"
clear_emulator_capabilities = 0
EOF

systemctl restart libvirtd

cat >/etc/hyper/config <<EOF
Kernel=/var/lib/hyper/kernel
Initrd=/var/lib/hyper/hyper-initrd.img
Bios=/var/lib/hyper/bios-qboot.bin
Cbfs=/var/lib/hyper/cbfs-qboot.rom
DisableIptables=true
StorageDriver=overlay
Hypervisor=libvirt
EOF

systemctl restart hyperd

# Build
cd $GOPATH/src/k8s.io
git clone https://github.com/feiskyer/kubernetes.git -b hyper-runtime
make clean && make all

# Bring up local cluster
CONTAINER_RUNTIME=hyper hack/local-up-cluster.sh

Run e2e test

# /bin/cp -f _output/local/bin/linux/amd64/* /usr/bin/
e2e.test --host="127.0.0.1:8080" --provider="local" --ginkgo.v=true --kubeconfig=/root/.kube/config --repo-root=$GOPATH/src/k8s.io/kubernetes -ginkgo.noColor -ginkgo.skip="SecurityContext|FSGrou|Port forwarding" -ginkgo.focus="Conformance"> /root/e2e.log

Summary

TIPs:

  • SecurityContext/FSGroup/Port-Forwarding is disabled since they are not supported in hyper runtime
  • Many emptydir volumes failed because of SeLinux, which is not supported for hyper runtime
Summarizing 22 Failures:

[Fail] [k8s.io] EmptyDir volumes [It] should support (root,0644,default) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

# Notes (feisky): limiting log bytes not supported in Hyper.
[Fail] [k8s.io] Kubectl client [k8s.io] Kubectl logs [It] should be able to retrieve and filter logs [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:837

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0777,default) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0644,default) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0666,default) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] Kubectl client [k8s.io] Guestbook application [It] should create and stop a working application [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1235

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0777,tmpfs) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] EmptyDir volumes [It] volume on default medium should have the correct mode [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0666,tmpfs) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] DNS [It] should provide DNS for the cluster [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/dns.go:229

[Fail] [k8s.io] EmptyDir volumes [It] should support (root,0644,tmpfs) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] Kubectl client [k8s.io] Proxy server [It] should support proxy with --port 0 [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1112

# Notes: Cluster dns is not setted up for local provider (it is set as 127.0.0.1)
# Docker runtime also failed on this case.
[Fail] [k8s.io] ClusterDns [Feature:Example] [It] should create pod that uses dns [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/example_cluster_dns.go:129

[Fail] [k8s.io] EmptyDir volumes [It] should support (root,0666,tmpfs) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] Pods [It] should be schedule with cpu and memory limits [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/pods.go:262

[Fail] [k8s.io] EmptyDir volumes [It] should support (root,0666,default) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] DNS [It] should provide DNS for services [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/dns.go:229

[Fail] [k8s.io] EmptyDir volumes [It] volume on tmpfs should have the correct mode [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] Pods [It] should be restarted with a docker exec "cat /tmp/health" liveness probe [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/pods.go:108

[Fail] [k8s.io] EmptyDir volumes [It] should support (root,0777,tmpfs) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] EmptyDir volumes [It] should support (root,0777,default) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

[Fail] [k8s.io] EmptyDir volumes [It] should support (non-root,0644,tmpfs) [Conformance] 
/root/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/util.go:1685

Ran 90 of 268 Specs in 3566.194 seconds
FAIL! -- 68 Passed | 22 Failed | 0 Pending | 178 Skipped --- FAIL: TestE2E (3566.23s)

Full logs is at e2e log

@resouer
Copy link

resouer commented Mar 15, 2016

First thoughts, for Port forwarding, if we does not need to support port forwarding in hyper side, we can skip these tests.

@resouer
Copy link

resouer commented Mar 15, 2016

As well as SecurityContext tests?

@feiskyer
Copy link
Author

feiskyer commented Mar 16, 2016

Y

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment