helm-acceptance-tests-results.md

$ make test-acceptance
CGO_ENABLED=0 /home/usr1/go/bin/gox -parallel=3 -output="_dist/{{.OS}}-{{.Arch}}/helm" -osarch='linux/amd64'  -tags '' -ldflags '-w -s -X helm.sh/helm/internal/version.gitCommit=821d7e6ef770c2b6705665ff4cfa39aedeef1b2a -X helm.sh/helm/internal/version.gitTreeState=dirty -extldflags "-static"' helm.sh/helm/cmd/helm
Number of parallel builds: 3

-->     linux/amd64: helm.sh/helm/cmd/helm
make[1]: Entering directory '/home/usr1/go/src/helm.sh/acceptance-testing'
+ REQUIRED_SYSTEM_COMMANDS=("kind" "kubectl" "python3" "pip" "virtualenv")
+ set +x
+++ dirname scripts/acceptance.sh
++ cd scripts
++ pwd
+ DIR=/home/usr1/go/src/helm.sh/acceptance-testing/scripts
+ cd /home/usr1/go/src/helm.sh/acceptance-testing/scripts/../
+ FINAL_DIR_NAME=.helm_acceptance_tests
+ ROBOT_PY_REQUIRES=robotframework==3.1.2
+ ROBOT_OUTPUT_DIR=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance
+ ROBOT_HELM_HOME_DIR=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests
+ ROBOT_VENV_DIR=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv
+ ROBOT_TEST_ROOT_DIR=/home/usr1/go/src/helm.sh/acceptance-testing
+ ROBOT_RUN_TESTS=.
+ ROBOT_RUN_TESTS=.
+ '[' '!' -z /home/usr1/go/src/helm.sh/helm/bin ']'
+ export PATH=/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ PATH=/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ export PATH=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv/bin:/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ PATH=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv/bin:/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ set +x
+ export XDG_CACHE_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/cache
+ XDG_CACHE_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/cache
+ mkdir -p /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/cache
+ export XDG_CONFIG_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/config
+ XDG_CONFIG_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/config
+ mkdir -p /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/config
+ export XDG_DATA_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/data
+ XDG_DATA_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/data
+ mkdir -p /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/data
+ set +x
====================
Running with Helm v3
====================
+ [[ ! -d /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv ]]
++ which python3
+ virtualenv -p /usr/bin/python3 /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv/bin/python3
Also creating executable in /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv/bin/python
Installing setuptools, pip, wheel...
done.
+ pip install robotframework==3.1.2
Collecting robotframework==3.1.2
  Downloading https://files.pythonhosted.org/packages/22/0f/1b9ffa0c4e59789b50e6034866e823b7d4a5c7eaedad7bfd0bba42f2aa9d/robotframework-3.1.2-py2.py3-none-any.whl (602kB)
     |████████████████████████████████| 604kB 367kB/s 
Installing collected packages: robotframework
Successfully installed robotframework-3.1.2
+ robot --outputdir=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance .
==============================================================================
Acceptance-Testing                                                            
==============================================================================
Acceptance-Testing.Kubernetes Versions :: Verify Helm functionality on mult...
==============================================================================
Helm works with Kubernetes 1.14.3                                     | FAIL |
Max time (60) reached waiting for cluster node
------------------------------------------------------------------------------
Helm works with Kubernetes 1.15.0                                     | FAIL |
Max time (60) reached waiting for cluster node
------------------------------------------------------------------------------
Acceptance-Testing.Kubernetes Versions :: Verify Helm functionalit... | FAIL |
2 critical tests, 0 passed, 2 failed
2 tests total, 0 passed, 2 failed
==============================================================================
Acceptance-Testing.Repos :: Verify helm repo commands work as expected.       
==============================================================================
No repos provisioned yet                                              | PASS |
------------------------------------------------------------------------------
Add a first valid repo                                                | PASS |
------------------------------------------------------------------------------
Add invalid repo without protocol                                     | PASS |
------------------------------------------------------------------------------
Add invalid repo with protocol                                        | PASS |
------------------------------------------------------------------------------
Add a second valid repo                                               | PASS |
------------------------------------------------------------------------------
Check output of repo list                                             | PASS |
------------------------------------------------------------------------------
Make sure both repos get updated                                      | PASS |
------------------------------------------------------------------------------
Try to remove inexistant repo                                         | PASS |
------------------------------------------------------------------------------
Remove a repo                                                         | PASS |
------------------------------------------------------------------------------
Make sure repo update will only update the remaining repo             | PASS |
------------------------------------------------------------------------------
Try removing an already removed repo                                  | PASS |
------------------------------------------------------------------------------
Remove last repo                                                      | PASS |
------------------------------------------------------------------------------
Check there are no more repos                                         | PASS |
------------------------------------------------------------------------------
Make sure repo update now fails, with a proper message                | PASS |
------------------------------------------------------------------------------
Acceptance-Testing.Repos :: Verify helm repo commands work as expe... | PASS |
14 critical tests, 14 passed, 0 failed
14 tests total, 14 passed, 0 failed
==============================================================================
Acceptance-Testing.Shells :: Verify Helm functionality on multiple shells.    
==============================================================================
Helm shell completion works                                           | FAIL |
Expected return code to be "0" but was "1".
------------------------------------------------------------------------------
Acceptance-Testing.Shells :: Verify Helm functionality on multiple... | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Acceptance-Testing                                                    | FAIL |
17 critical tests, 14 passed, 3 failed
17 tests total, 14 passed, 3 failed
==============================================================================
Output:  /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/output.xml
Log:     /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/log.html
Report:  /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/report.html
Makefile:5: recipe for target 'acceptance' failed
make[1]: *** [acceptance] Error 3
make[1]: Leaving directory '/home/usr1/go/src/helm.sh/acceptance-testing'
Makefile:91: recipe for target 'test-acceptance' failed
make: *** [test-acceptance] Error 2

$ make test-acceptance-completion
CGO_ENABLED=0 /home/usr1/go/bin/gox -parallel=3 -output="_dist/{{.OS}}-{{.Arch}}/helm" -osarch='linux/amd64'  -tags '' -ldflags '-w -s -X helm.sh/helm/internal/version.gitCommit=821d7e6ef770c2b6705665ff4cfa39aedeef1b2a -X helm.sh/helm/internal/version.gitTreeState=dirty -extldflags "-static"' helm.sh/helm/cmd/helm
Number of parallel builds: 3

-->     linux/amd64: helm.sh/helm/cmd/helm
make[1]: Entering directory '/home/usr1/go/src/helm.sh/acceptance-testing'
+ REQUIRED_SYSTEM_COMMANDS=("kind" "kubectl" "python3" "pip" "virtualenv")
+ set +x
+++ dirname scripts/acceptance.sh
++ cd scripts
++ pwd
+ DIR=/home/usr1/go/src/helm.sh/acceptance-testing/scripts
+ cd /home/usr1/go/src/helm.sh/acceptance-testing/scripts/../
+ FINAL_DIR_NAME=.helm_acceptance_tests
+ ROBOT_PY_REQUIRES=robotframework==3.1.2
+ ROBOT_OUTPUT_DIR=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance
+ ROBOT_HELM_HOME_DIR=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests
+ ROBOT_VENV_DIR=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv
+ ROBOT_TEST_ROOT_DIR=/home/usr1/go/src/helm.sh/acceptance-testing
+ ROBOT_RUN_TESTS=shells.robot
+ ROBOT_RUN_TESTS=shells.robot
+ '[' '!' -z /home/usr1/go/src/helm.sh/helm/bin ']'
+ export PATH=/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ PATH=/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ export PATH=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv/bin:/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ PATH=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv/bin:/home/usr1/go/src/helm.sh/helm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/usr1/go/bin:/home/usr1/.kubeadm-dind-cluster
+ set +x
+ export XDG_CACHE_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/cache
+ XDG_CACHE_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/cache
+ mkdir -p /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/cache
+ export XDG_CONFIG_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/config
+ XDG_CONFIG_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/config
+ mkdir -p /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/config
+ export XDG_DATA_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/data
+ XDG_DATA_HOME=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/data
+ mkdir -p /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.helm_acceptance_tests/data
+ set +x
====================
Running with Helm v3
====================
+ [[ ! -d /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/.venv ]]
+ robot --outputdir=/home/usr1/go/src/helm.sh/acceptance-testing/.acceptance shells.robot
==============================================================================
Shells :: Verify Helm functionality on multiple shells.                       
==============================================================================
Helm shell completion works                                           | FAIL |
Expected return code to be "0" but was "1".
------------------------------------------------------------------------------
Shells :: Verify Helm functionality on multiple shells.               | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output:  /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/output.xml
Log:     /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/log.html
Report:  /home/usr1/go/src/helm.sh/acceptance-testing/.acceptance/report.html
Makefile:5: recipe for target 'acceptance' failed
make[1]: *** [acceptance] Error 1
make[1]: Leaving directory '/home/usr1/go/src/helm.sh/acceptance-testing'
Makefile:91: recipe for target 'test-acceptance' failed
make: *** [test-acceptance] Error 2