Created
October 14, 2020 16:14
-
-
Save teutat3s/d7aa7374bf214a507299d366d61d2615 to your computer and use it in GitHub Desktop.
Nomad remote taskdriver failing tests log
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$ sudo -E PATH=$(pwd)/bin:$PATH make test | |
| make[1]: Entering directory '/opt/gopath/src/github.com/hashicorp/nomad' | |
| --> Making [GH-xxxx] references clickable... | |
| --> Formatting HCL | |
| ==> Removing old development build... | |
| ==> Building pkg/linux_amd64/nomad with tags codegen_generated ... | |
| ==> Running Nomad test suites: | |
| gotestsum -- \ | |
| \ | |
| -cover \ | |
| -timeout=15m \ | |
| -tags "codegen_generated" \ | |
| "./..." | |
| ✓ acl (cached) (coverage: 84.1% of statements) | |
| ✓ . (cached) (coverage: 1.7% of statements) | |
| ✓ client/allocdir (cached) (coverage: 61.6% of statements) | |
| ✓ client/allochealth (cached) (coverage: 57.2% of statements) | |
| ✓ client/allocrunner (cached) (coverage: 66.7% of statements) | |
| ✓ client/allocrunner/taskrunner/getter (cached) (coverage: 84.2% of statements) | |
| ✓ client/allocrunner/taskrunner/restarts (cached) (coverage: 78.7% of statements) | |
| ✓ client/allocrunner/taskrunner/template (cached) (coverage: 85.5% of statements) | |
| ✓ client/config (cached) (coverage: 5.0% of statements) | |
| ✓ client/allocwatcher (cached) (coverage: 42.7% of statements) | |
| ✓ client/consul (cached) (coverage: 9.5% of statements) | |
| ✓ client/dynamicplugins (cached) (coverage: 74.2% of statements) | |
| ✓ client/devicemanager (cached) (coverage: 69.5% of statements) | |
| ✓ client/lib/fifo (cached) (coverage: 83.3% of statements) | |
| ✓ client/fingerprint (cached) (coverage: 74.6% of statements) | |
| ✓ client/lib/streamframer (cached) (coverage: 89.7% of statements) | |
| ✓ client/logmon/logging (cached) (coverage: 75.6% of statements) | |
| ✓ client/pluginmanager (cached) (coverage: 45.2% of statements) | |
| ✓ client/pluginmanager/csimanager (cached) (coverage: 81.8% of statements) | |
| ✓ client/pluginmanager/drivermanager (cached) (coverage: 55.4% of statements) | |
| ✓ client/servers (cached) (coverage: 80.4% of statements) | |
| ✓ client/state (cached) (coverage: 72.2% of statements) | |
| ✓ client/stats (cached) (coverage: 81.0% of statements) | |
| ✓ client/structs (cached) (coverage: 0.7% of statements) | |
| ✓ client/taskenv (cached) (coverage: 91.0% of statements) | |
| ✓ client/vaultclient (cached) (coverage: 54.1% of statements) | |
| ✓ client/allocrunner/taskrunner (cached) (coverage: 69.6% of statements) | |
| ✓ client/logmon (10.511s) (coverage: 63.0% of statements) | |
| ✓ command/agent/consul (cached) (coverage: 76.3% of statements) | |
| ✓ command/agent/host (cached) (coverage: 90.0% of statements) | |
| ✓ command/agent/monitor (cached) (coverage: 81.4% of statements) | |
| ✓ command/agent/pprof (cached) (coverage: 86.1% of statements) | |
| ✓ devices/gpu/nvidia (cached) (coverage: 75.7% of statements) | |
| ✓ devices/gpu/nvidia/nvml (cached) (coverage: 50.0% of statements) | |
| ✓ drivers/docker (cached) (coverage: 63.9% of statements) | |
| ✓ drivers/docker/docklog (cached) (coverage: 38.1% of statements) | |
| ✓ client (cached) (coverage: 74.3% of statements) | |
| ∅ client/allocdir/input | |
| ∅ client/allocrunner/interfaces | |
| ∅ client/allocrunner/state | |
| ∅ client/allocrunner/taskrunner/interfaces | |
| ∅ client/allocrunner/taskrunner/state | |
| ∅ client/devicemanager/state | |
| ∅ client/interfaces | |
| ∅ client/lib/nsutil | |
| ∅ client/logmon/proto | |
| ∅ client/pluginmanager/drivermanager/state | |
| ∅ client/testutil | |
| ✓ command (27.931s) (coverage: 45.5% of statements) | |
| ✓ drivers/mock (cached) (coverage: 1.1% of statements) | |
| ✓ drivers/java (cached) (coverage: 58.0% of statements) | |
| ✓ drivers/qemu (cached) (coverage: 55.8% of statements) | |
| ✓ drivers/rawexec (cached) (coverage: 68.4% of statements) | |
| ✓ drivers/shared/eventer (cached) (coverage: 65.9% of statements) | |
| ✓ drivers/shared/resolvconf (cached) (coverage: 27.0% of statements) | |
| ✓ e2e (cached) | |
| ✓ e2e/connect (cached) (coverage: 2.0% of statements) | |
| ✓ e2e/migrations (cached) | |
| ✓ e2e/rescheduling (cached) | |
| ✓ e2e/vault (cached) | |
| ✓ helper (cached) (coverage: 31.7% of statements) | |
| ✓ helper/args (cached) (coverage: 87.5% of statements) | |
| ✓ helper/boltdd (cached) (coverage: 80.3% of statements) | |
| ✓ helper/constraints/semver (cached) (coverage: 97.2% of statements) | |
| ✓ helper/escapingio (cached) (coverage: 100.0% of statements) | |
| ✓ helper/fields (cached) (coverage: 62.7% of statements) | |
| ✓ helper/flag-helpers (cached) (coverage: 9.5% of statements) | |
| ✓ helper/flatmap (cached) (coverage: 78.3% of statements) | |
| ✓ helper/freeport (cached) (coverage: 81.7% of statements) | |
| ✓ helper/gated-writer (cached) (coverage: 100.0% of statements) | |
| ✓ helper/pluginutils/hclspecutils (cached) (coverage: 79.6% of statements) | |
| ✓ helper/pluginutils/hclutils (cached) (coverage: 82.9% of statements) | |
| ✓ helper/pluginutils/loader (cached) (coverage: 77.1% of statements) | |
| ✓ helper/pluginutils/singleton (cached) (coverage: 92.9% of statements) | |
| ✓ helper/pool (cached) (coverage: 30.7% of statements) | |
| ✓ helper/raftutil (cached) (coverage: 11.7% of statements) | |
| ✓ helper/snapshot (cached) (coverage: 76.4% of statements) | |
| ✓ helper/tlsutil (cached) (coverage: 81.4% of statements) | |
| ✓ helper/useragent (cached) (coverage: 50.0% of statements) | |
| ✓ helper/uuid (cached) (coverage: 75.0% of statements) | |
| ✓ drivers/shared/executor (cached) (coverage: 42.4% of statements) | |
| ✓ jobspec (35ms) (coverage: 76.1% of statements) | |
| ✓ lib/circbufwriter (cached) (coverage: 91.7% of statements) | |
| ✓ lib/delayheap (cached) (coverage: 67.9% of statements) | |
| ✓ lib/kheap (cached) (coverage: 70.8% of statements) | |
| ✓ internal/testing/apitests (6.024s) | |
| ✓ nomad/deploymentwatcher (cached) (coverage: 81.5% of statements) | |
| ✓ nomad/drainer (cached) (coverage: 59.4% of statements) | |
| ✓ nomad/state (cached) (coverage: 74.3% of statements) | |
| ✓ nomad/structs (cached) (coverage: 3.9% of statements) | |
| ✓ nomad/structs/config (cached) (coverage: 73.7% of statements) | |
| ✓ nomad/volumewatcher (cached) (coverage: 86.8% of statements) | |
| ✓ plugins/base (cached) (coverage: 64.5% of statements) | |
| ✓ plugins/csi (cached) (coverage: 63.3% of statements) | |
| ✓ plugins/device (cached) (coverage: 58.9% of statements) | |
| ✓ plugins/drivers (cached) (coverage: 3.9% of statements) | |
| ✓ plugins/drivers/testutils (cached) (coverage: 7.8% of statements) | |
| ✓ plugins/shared/structs (cached) (coverage: 48.9% of statements) | |
| ✓ drivers/exec (cached) (coverage: 63.4% of statements) | |
| ✓ testutil (cached) (coverage: 0.0% of statements) | |
| ✓ command/agent (1m0.825s) (coverage: 70.1% of statements) | |
| ∅ command/agent/event | |
| ∅ command/raft_tools | |
| ∅ demo/digitalocean/app | |
| ∅ devices/gpu/nvidia/cmd | |
| ∅ drivers/docker/cmd | |
| ∅ drivers/docker/docklog/proto | |
| ∅ drivers/docker/util | |
| ∅ drivers/shared/executor/proto | |
| ∅ e2e/affinities | |
| ∅ e2e/cli | |
| ∅ e2e/cli/command | |
| ∅ e2e/clientstate | |
| ∅ e2e/consul | |
| ∅ e2e/consulacls | |
| ∅ e2e/consultemplate | |
| ∅ e2e/csi | |
| ∅ e2e/deployment | |
| ∅ e2e/e2eutil | |
| ∅ e2e/example | |
| ∅ e2e/execagent | |
| ∅ e2e/framework | |
| ∅ e2e/framework/provisioning | |
| ∅ e2e/hostvolumes | |
| ∅ e2e/lifecycle | |
| ∅ e2e/metrics | |
| ∅ e2e/nomad09upgrade | |
| ∅ e2e/nomadexec | |
| ∅ e2e/podman | |
| ∅ e2e/spread | |
| ∅ e2e/systemsched | |
| ∅ e2e/taskevents | |
| ∅ helper/codec | |
| ∅ helper/discover | |
| ∅ helper/grpc-middleware/logging | |
| ∅ helper/logging | |
| ∅ helper/mount | |
| ∅ helper/noxssrw | |
| ∅ helper/pluginutils/catalog | |
| ∅ helper/pluginutils/grpcutils | |
| ∅ helper/stats | |
| ∅ helper/testlog | |
| ∅ helper/testtask | |
| ∅ helper/winsvc | |
| ✖ scheduler (25.754s) (coverage: 88.8% of statements) | |
| ✖ nomad (2m21.928s) (coverage: 76.2% of statements) | |
| ∅ nomad/mock | |
| ∅ nomad/types | |
| ∅ plugins | |
| ∅ plugins/base/proto | |
| ∅ plugins/base/structs | |
| ∅ plugins/csi/fake | |
| ∅ plugins/csi/testing | |
| ∅ plugins/device/cmd/example | |
| ∅ plugins/device/cmd/example/cmd | |
| ∅ plugins/device/proto | |
| ∅ plugins/drivers/proto | |
| ∅ plugins/drivers/utils | |
| ∅ plugins/shared/cmd/launcher | |
| ∅ plugins/shared/cmd/launcher/command | |
| ∅ plugins/shared/hclspec | |
| ∅ plugins/shared/structs/proto | |
| ∅ version | |
| === Skipped | |
| === SKIP: client/allocdir TestLinuxUnprivilegedSecretDir (0.00s) | |
| fs_linux_test.go:113: Must not be run as root | |
| === SKIP: client/allocdir TestTaskDir_NonRoot_Image (0.00s) | |
| task_dir_test.go:91: test should be run as non-root user | |
| === SKIP: client/allocdir TestTaskDir_NonRoot (0.00s) | |
| task_dir_test.go:114: test should be run as non-root user | |
| === SKIP: client/allocrunner/taskrunner TestSIDSHook_recoverToken_unReadable (0.00s) | |
| sids_hook_test.go:98: test only works as non-root | |
| === SKIP: client/allocrunner/taskrunner TestSIDSHook_writeToken_unWritable (0.00s) | |
| sids_hook_test.go:145: test only works as non-root | |
| === SKIP: client/allocrunner/taskrunner TestTaskRunner_DeriveSIToken_UnWritableTokenFile (0.00s) | |
| sids_hook_test.go:273: test only works as non-root | |
| === SKIP: client/allocrunner/taskrunner TestEnvoyBootstrapHook_maybeLoadSIToken (0.00s) | |
| === PAUSE TestEnvoyBootstrapHook_maybeLoadSIToken | |
| === CONT TestEnvoyBootstrapHook_maybeLoadSIToken | |
| envoybootstrap_hook_test.go:52: test only works as non-root | |
| === SKIP: client/pluginmanager/csimanager TestVolumeManager_ensureStagingDir/Returns_positive_mount_info (0.00s) | |
| === SKIP: drivers/docker TestDockerDriver_AdvertiseIPv6Address (0.02s) | |
| === PAUSE TestDockerDriver_AdvertiseIPv6Address | |
| === CONT TestDockerDriver_AdvertiseIPv6Address | |
| 2020-10-14T15:51:34.506Z [TRACE] eventer/eventer.go:68: docker: task event loop shutdown | |
| docker.go:36: Successfully connected to docker daemon running version 19.03.13 | |
| docker.go:36: Successfully connected to docker daemon running version 19.03.13 | |
| driver_test.go:2466: IPv6 not enabled on bridge network, skipping | |
| === SKIP: drivers/exec TestExecDriver_Fingerprint_NonLinux (0.00s) | |
| === PAUSE TestExecDriver_Fingerprint_NonLinux | |
| === CONT TestExecDriver_Fingerprint_NonLinux | |
| driver_test.go:59: Test only available not on Linux | |
| === SKIP: e2e TestE2E (0.00s) | |
| e2e_test.go:32: Skipping e2e tests, NOMAD_E2E not set | |
| === SKIP: e2e/migrations TestJobMigrations (0.00s) | |
| migrations_test.go:218: skipping test in non-integration mode. | |
| === SKIP: e2e/migrations TestMigrations_WithACLs (0.00s) | |
| migrations_test.go:269: skipping test in non-integration mode. | |
| === SKIP: e2e/rescheduling TestServerSideRestarts (0.00s) | |
| server_side_restarts_suite_test.go:16: skipping test in non-integration mode. | |
| === SKIP: e2e/vault TestVaultCompatibility (0.00s) | |
| vault_test.go:304: skipping test in non-integration mode: add -integration flag to run | |
| === SKIP: helper/tlsutil TestConfig_outgoingWrapper_BadCert (0.00s) | |
| === SKIP: nomad TestAutopilot_CleanupStaleRaftServer (0.00s) | |
| autopilot_test.go:252: TestAutopilot_CleanupDeadServer is very flaky, removing it for now | |
| === SKIP: nomad/structs TestNetworkIndex_Overcommitted (0.00s) | |
| network_test.go:13: | |
| === SKIP: scheduler TestBinPackIterator_Network_Failure (0.00s) | |
| rank_test.go:377: | |
| === Failed | |
| === FAIL: nomad TestEvalBroker_Wait (0.02s) | |
| === PAUSE TestEvalBroker_Wait | |
| === CONT TestEvalBroker_Wait | |
| eval_broker_test.go:1225: bad: &nomad.BrokerStats{TotalReady:0, TotalUnacked:0, TotalBlocked:0, TotalWaiting:1, ByScheduler:map[string]*nomad.SchedulerStats{}} | |
| === FAIL: scheduler TestServiceSched_Migrate_CanaryStatus (0.00s) | |
| 2020-10-14T16:12:50.175Z [TRACE] scheduler/rank.go:175: service_sched.binpack: NewBinPackIterator created: eval_id=a1c160ec-a2dd-a24a-8972-efe596045272 job_id=mock-service-967e74be-54fd-7d6a-440d-e8cf61eba77c namespace=default algorithm=binpack | |
| 2020-10-14T16:12:50.175Z [DEBUG] scheduler/generic_sched.go:357: service_sched: reconciled current state with desired state: eval_id=a1c160ec-a2dd-a24a-8972-efe596045272 job_id=mock-service-967e74be-54fd-7d6a-440d-e8cf61eba77c namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) | |
| Created Deployment: "20650f48-bc43-eab0-da2e-2b00b99cbf37" | |
| Desired Changes for "web": (place 0) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 3) (canary 1)" | |
| 2020-10-14T16:12:50.175Z [DEBUG] scheduler/util.go:535: service_sched: setting eval status: eval_id=a1c160ec-a2dd-a24a-8972-efe596045272 job_id=mock-service-967e74be-54fd-7d6a-440d-e8cf61eba77c namespace=default status=complete | |
| 2020-10-14T16:12:50.175Z [TRACE] scheduler/rank.go:175: service_sched.binpack: NewBinPackIterator created: eval_id=a1c160ec-a2dd-a24a-8972-efe596045272 job_id=mock-service-967e74be-54fd-7d6a-440d-e8cf61eba77c namespace=default algorithm=binpack | |
| 2020-10-14T16:12:50.175Z [DEBUG] scheduler/generic_sched.go:357: service_sched: reconciled current state with desired state: eval_id=a1c160ec-a2dd-a24a-8972-efe596045272 job_id=mock-service-967e74be-54fd-7d6a-440d-e8cf61eba77c namespace=default results="Total changes: (place 3) (destructive 0) (inplace 0) (stop 4) | |
| Desired Changes for "web": (place 3) (inplace 0) (destructive 0) (stop 3) (migrate 0) (ignore 0) (canary 0)" | |
| 2020-10-14T16:12:50.176Z [DEBUG] scheduler/util.go:535: service_sched: setting eval status: eval_id=a1c160ec-a2dd-a24a-8972-efe596045272 job_id=mock-service-967e74be-54fd-7d6a-440d-e8cf61eba77c namespace=default status=complete | |
| generic_sched_test.go:5570: | |
| Error Trace: generic_sched_test.go:5570 | |
| Error: Not equal: | |
| expected: 0x0 | |
| actual : 0x1 | |
| Test: TestServiceSched_Migrate_CanaryStatus | |
| === FAIL: scheduler TestReconciler_LostNode_ScaleUp (0.00s) | |
| reconcile_test.go:819: Index 0 repeated too many times | |
| All names: [mock-service-aab14370-f2fb-1870-5f89-519ef31eb03e.web[0] mock-service-aab14370-f2fb-1870-5f89-519ef31eb03e.web[1] mock-service-aab14370-f2fb-1870-5f89-519ef31eb03e.web[0] mock-service-aab14370-f2fb-1870-5f89-519ef31eb03e.web[1] mock-service-aab14370-f2fb-1870-5f89-519ef31eb03e.web[10] mock-service-aab14370-f2fb-1870-5f89-519ef31eb03e.web[11] mock-service-aab14370-f2fb-1870-5f89-519ef31eb03e.web[12]] | |
| === FAIL: scheduler TestReconciler_LostNode_ScaleDown (0.00s) | |
| reconcile_test.go:853: | |
| Error Trace: reconcile_test.go:281 | |
| reconcile_test.go:853 | |
| Error: "[{mock-service-a32c5402-6abe-7d75-b126-74aaef9aab07.web[0] %!s(bool=false) %!s(*structs.TaskGroup=&{web 5 <nil> 0xc0000ed1a0 [] <nil> 0xc0000ed200 [0xc00007bb80] 0xc00031aa80 map[elb_check_interval:30s elb_check_min:3 elb_check_type:http] 0xc0001fddc0 [] [] [] [] map[] <nil> <nil>}) %!s(*structs.Allocation=&{false 4e8ed269-1f7d-413c-1bfb-63cccd32f99a default f4eeb748-ddcc-b1b0-16a8-8d3f23353601 mock-service-a32c5402-6abe-7d75-b126-74aaef9aab07.web[0] bee399df-0462-0961-501e-f1428110e334 mock-service-a32c5402-6abe-7d75-b126-74aaef9aab07 0xc0000f0480 web 0xc0000aad70 0xc0000aadc0 map[web:0xc0000aacd0] 0xc0000aae10 <nil> run {<nil> <nil> <nil>} pending map[] [] <nil> <nil> [] 0 0 0 0 0}) %!s(bool=false) %!s(bool=false) %!s(uint64=0) %!s(bool=true)} {mock-service-a32c5402-6abe-7d75-b126-74aaef9aab07.web[1] %!s(bool=false) %!s(*structs.TaskGroup=&{web 5 <nil> 0xc0000ed1a0 [] <nil> 0xc0000ed200 [0xc00007bb80] 0xc00031aa80 map[elb_check_interval:30s elb_check_min:3 elb_check_type:http] 0xc0001fddc0 [] [] [] [] map[] <nil> <nil>}) %!s(*structs.Allocation=&{false 7c39976c-d144-ee94-1da8-0ee31dfd434a default ff854c6a-bf30-f850-8122-306e8b0186dd mock-service-a32c5402-6abe-7d75-b126-74aaef9aab07.web[1] 611e1967-4134-fd6e-95a6-94773d8f6e45 mock-service-a32c5402-6abe-7d75-b126-74aaef9aab07 0xc0000f0480 web 0xc0000aaf00 0xc0000aafa0 map[web:0xc0000aae60] 0xc0000ab040 <nil> run {<nil> <nil> <nil>} pending map[] [] <nil> <nil> [] 0 0 0 0 0}) %!s(bool=false) %!s(bool=false) %!s(uint64=0) %!s(bool=true)}]" should have 0 item(s), but has 2 | |
| Test: TestReconciler_LostNode_ScaleDown | |
| Messages: Expected Placements | |
| reconcile_test.go:853: | |
| Error Trace: reconcile_test.go:286 | |
| reconcile_test.go:853 | |
| Error: Not equal: | |
| expected: map[string]*structs.DesiredUpdates{"web":(place 0) (inplace 0) (destructive 0) (stop 5) (migrate 0) (ignore 5) (canary 0)} | |
| actual : map[string]*structs.DesiredUpdates{"web":(place 2) (inplace 0) (destructive 0) (stop 5) (migrate 0) (ignore 5) (canary 0)} | |
| Diff: | |
| --- Expected | |
| +++ Actual | |
| @@ -3,3 +3,3 @@ | |
| Ignore: (uint64) 5, | |
| - Place: (uint64) 0, | |
| + Place: (uint64) 2, | |
| Migrate: (uint64) 0, | |
| Test: TestReconciler_LostNode_ScaleDown | |
| Messages: Expected Desired TG Update Annotations | |
| DONE 4647 tests, 19 skipped, 4 failures in 204.155s | |
| GNUmakefile:327: recipe for target 'test-nomad' failed | |
| make[1]: *** [test-nomad] Error 1 | |
| make[1]: Leaving directory '/opt/gopath/src/github.com/hashicorp/nomad' | |
| GNUmakefile:312: recipe for target 'test' failed | |
| make: *** [test] Error 2 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment