Skip to content

Instantly share code, notes, and snippets.

@atombender

atombender/workflow-controller.log Secret

Created July 19, 2021 14:50
Show Gist options
  • Save atombender/03520de681ca7a09c9535065469ddc22 to your computer and use it in GitHub Desktop.
Save atombender/03520de681ca7a09c9535065469ddc22 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
workflow-controller.log
time="2021-07-19T14:49:19Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2021-07-19T14:49:19Z" level=info msg="cron config" cronSyncPeriod=10s
time="2021-07-19T14:49:19.966Z" level=info msg="config map" name=workflow-controller-configmap
time="2021-07-19T14:49:19.975Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:19.982Z" level=info msg="Configuration:\nartifactRepository:\n archiveLogs: true\n gcs:\n bucket: sanity-staging-argo-artifacts\n serviceAccountKeySecret:\n key: serviceAccountKey\n name: google-gcs-credentials\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npersistence:\n clusterName: staging\n connectionPool:\n maxIdleConns: 100\n postgresql:\n database: argo\n host: 10.132.0.20\n passwordSecret:\n key: password\n name: argo-postgres-config\n port: 5432\n tableName: argo_workflows\n userNameSecret:\n key: username\n name: argo-postgres-config\npodSpecLogStrategy: {}\ntelemetryConfig: {}\nworkflowDefaults:\n metadata:\n annotations:\n argo: workflows\n creationTimestamp: null\n labels:\n cluster: staging\n spec:\n arguments: {}\n parallelism: 3\n ttlStrategy:\n secondsAfterSuccess: 5\n status:\n finishedAt: null\n startedAt: null\n"
time="2021-07-19T14:49:19.982Z" level=info msg="Persistence configuration enabled"
time="2021-07-19T14:49:19.982Z" level=info msg="Creating DB session"
time="2021-07-19T14:49:19.985Z" level=info msg="Get secrets 200"
time="2021-07-19T14:49:19.990Z" level=info msg="Get secrets 200"
time="2021-07-19T14:49:20.003Z" level=info msg="Persistence Session created successfully"
time="2021-07-19T14:49:20.004Z" level=info msg="Migrating database schema" clusterName=staging dbType=postgres
time="2021-07-19T14:49:20.097Z" level=info msg="Node status offloading is disabled"
time="2021-07-19T14:49:20.097Z" level=info msg="Workflow archiving is disabled"
time="2021-07-19T14:49:20.098Z" level=info msg="Starting Workflow Controller" version=untagged
time="2021-07-19T14:49:20.098Z" level=info msg="Workers: workflow: 32, pod: 32, pod cleanup: 4"
time="2021-07-19T14:49:20.101Z" level=info msg="Watch configmaps 200"
time="2021-07-19T14:49:20.101Z" level=info msg="List configmaps 200"
time="2021-07-19T14:49:20.107Z" level=info msg="List workflowtemplates 200"
time="2021-07-19T14:49:20.109Z" level=info msg="Watch configmaps 200"
time="2021-07-19T14:49:20.116Z" level=info msg="List pods 200"
time="2021-07-19T14:49:20.132Z" level=info msg="Watch workflowtemplates 200"
time="2021-07-19T14:49:20.151Z" level=info msg="Watch pods 200"
time="2021-07-19T14:49:20.206Z" level=info msg="List workflows 200"
time="2021-07-19T14:49:20.358Z" level=info msg="Watch workflows 200"
time="2021-07-19T14:49:20.502Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-19T14:49:20.504Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-19T14:49:20.507Z" level=info msg="Create selfsubjectaccessreviews 201"
time="2021-07-19T14:49:20.510Z" level=info msg="List clusterworkflowtemplates 200"
time="2021-07-19T14:49:20.510Z" level=info msg="List workflows 200"
time="2021-07-19T14:49:20.511Z" level=info msg="Watch clusterworkflowtemplates 200"
time="2021-07-19T14:49:20.521Z" level=info msg="Manager initialized successfully"
I0719 14:49:20.522003 1 leaderelection.go:243] attempting to acquire leader lease argo/workflow-controller...
time="2021-07-19T14:49:20.522Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
time="2021-07-19T14:49:20.525Z" level=info msg="Get leases 200"
time="2021-07-19T14:49:20.532Z" level=info msg="Update leases 200"
I0719 14:49:20.532557 1 leaderelection.go:253] successfully acquired lease argo/workflow-controller
time="2021-07-19T14:49:20.532Z" level=info msg="started leading" id=workflow-controller-599ff5679b-vn6dq
time="2021-07-19T14:49:20.538Z" level=info msg="Get leases 200"
time="2021-07-19T14:49:20.532Z" level=info msg="new leader" id=workflow-controller-599ff5679b-vn6dq leader=workflow-controller-599ff5679b-vn6dq
time="2021-07-19T14:49:20.538Z" level=info msg="Performing periodic GC every 5m0s"
time="2021-07-19T14:49:20.538Z" level=info msg="Archive disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
time="2021-07-19T14:49:20.539Z" level=info msg="Starting workflow TTL controller (workflowTTLWorkers 4)"
time="2021-07-19T14:49:20.539Z" level=info msg="Started workflow TTL worker"
time="2021-07-19T14:49:20.539Z" level=info msg="Starting CronWorkflow controller"
time="2021-07-19T14:49:20.539Z" level=info msg="Processing workflow" namespace=e2e workflow=test-webhooks-5frj8
time="2021-07-19T14:49:20.540Z" level=info msg="Processing workflow" namespace=e2e workflow=webhook-test-q2bc7
time="2021-07-19T14:49:20.542Z" level=info msg="Queueing Failed workflow e2e/test-gantry-nxstx for delete in 23h10m42s"
time="2021-07-19T14:49:20.545Z" level=info msg="Queueing Succeeded workflow gradient-perfsuite/gradient-perfsuite-wvjnc for delete in 1h14m39s"
time="2021-07-19T14:49:20.552Z" level=info msg="Processing workflow" namespace=e2e workflow=test-webhooks-6rrzs
time="2021-07-19T14:49:20.553Z" level=info msg="List workflows 200"
time="2021-07-19T14:49:20.554Z" level=info msg="Update leases 200"
time="2021-07-19T14:49:20.555Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:20.555Z" level=info msg="Create events 201"
time="2021-07-19T14:49:20.555Z" level=info msg="Watch workflows 200"
time="2021-07-19T14:49:20.556Z" level=info msg="Processing workflow" namespace=e2e workflow=test-webhooks-2bwkk
time="2021-07-19T14:49:20.556Z" level=info msg="Processing workflow" namespace=e2e workflow=test-webhooks-9ms6g
time="2021-07-19T14:49:20.559Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:20.559Z" level=info msg="e2e/ConfigMap/semaphore/all-workflows acquired by e2e/test-webhooks-5frj8 " semaphore=e2e/ConfigMap/semaphore/all-workflows
time="2021-07-19T14:49:20.561Z" level=info msg="Queueing Failed workflow e2e/test-gantry-gm72l for delete in 22h40m5s"
time="2021-07-19T14:49:20.562Z" level=info msg="Running OnExit handler: exit-handler" namespace=e2e workflow=test-webhooks-5frj8
time="2021-07-19T14:49:20.562Z" level=info msg="Checking daemoned children of " namespace=e2e workflow=test-webhooks-5frj8
time="2021-07-19T14:49:20.563Z" level=info msg="List cronworkflows 200"
time="2021-07-19T14:49:20.567Z" level=info msg="Queueing Failed workflow e2e/test-gantry-lxr6d for delete in 22h25m21s"
time="2021-07-19T14:49:20.569Z" level=info msg="Queueing Error workflow gradient-perfsuite/gradient-perfsuite-492wn for delete in 22h58m57s"
time="2021-07-19T14:49:20.569Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:20.569Z" level=warning msg="Failed to acquire the lock" namespace=e2e workflow=webhook-test-q2bc7
time="2021-07-19T14:49:20.569Z" level=info msg="Checking daemoned children of " namespace=e2e workflow=webhook-test-q2bc7
time="2021-07-19T14:49:20.570Z" level=info msg="Queueing Succeeded workflow gradient-perfsuite/gradient-perfsuite-5xqvn for delete in 20h56m38s"
time="2021-07-19T14:49:20.571Z" level=info msg="Processing workflow" namespace=e2e workflow=webhook-test-mg7vj
time="2021-07-19T14:49:20.572Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:20.572Z" level=info msg="Watch cronworkflows 200"
time="2021-07-19T14:49:20.572Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=e2e workflow=test-webhooks-6rrzs
time="2021-07-19T14:49:20.572Z" level=panic msg="workflow is already fulfilled" fromPhase=Failed namespace=e2e toPhase=Pending workflow=test-webhooks-6rrzs
time="2021-07-19T14:49:20.572Z" level=error msg="Recovered from panic" namespace=e2e r="&{0xc0000f9c70 map[fromPhase:Failed namespace:e2e toPhase:Pending workflow:test-webhooks-6rrzs] 2021-07-19 14:49:20.572684792 +0000 UTC m=+0.640629945 panic <nil> workflow is already fulfilled <nil> <nil> }" stack="goroutine 223 [running]:\nruntime/debug.Stack(0xc0016df689, 0x1e9f060, 0xc0000eca80)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9f\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2(0xc0019a53f0, 0x2198d40, 0xc000128000)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:185 +0xd1\npanic(0x1e9f060, 0xc0000eca80)\n\t/usr/local/go/src/runtime/panic.go:975 +0x47a\ngithub.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc001d33440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345\ngithub.com/sirupsen/logrus.(*Entry).Log(0xc00019e700, 0x0, 0xc000d86ba8, 0x1, 0x1)\n\t/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0\ngithub.com/sirupsen/logrus.(*Entry).Panic(0xc00019e700, 0xc000d86ba8, 0x1, 0x1)\n\t/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0019a53f0, 0x2198d40, 0xc000128000, 0x1ed7572, 0x7, 0xc00046e280, 0x49)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0xc0019a53f0, 0x2198d40, 0xc000128000)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:233 +0x3ca5\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0xc000100c00, 0x2198d40, 0xc000128000, 0x0)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:754 +0x7fb\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0xc000100c00)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:675 +0x9b\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00146b180)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:155 +0x5f\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00146b180, 0x214a840, 0xc00096a660, 0x2020202020202001, 0xc000ca6480)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:156 +0xad\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00146b180, 0x3b9aca00, 0x0, 0x65706572225c2001, 0xc000ca6480)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:133 +0x98\nk8s.io/apimachinery/pkg/util/wait.Until(0xc00146b180, 0x3b9aca00, 0xc000ca6480)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:90 +0x4d\ncreated by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).startLeading\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:298 +0x48a\n" workflow=test-webhooks-6rrzs
time="2021-07-19T14:49:20.573Z" level=info msg="Queueing Failed workflow e2e/test-gantry-l64l9 for delete in 23h54m17s"
time="2021-07-19T14:49:20.573Z" level=panic msg="workflow is already fulfilled" fromPhase=Failed namespace=e2e toPhase=Error workflow=test-webhooks-6rrzs
time="2021-07-19T14:49:20.573Z" level=info msg="Checking daemoned children of " namespace=e2e workflow=test-webhooks-6rrzs
time="2021-07-19T14:49:20.575Z" level=info msg="Queueing Failed workflow e2e/test-webhooks-lv6x6 for delete in 22h17m16s"
time="2021-07-19T14:49:20.575Z" level=info msg="Queueing Failed workflow e2e/test-gantry-j4xr7 for delete in 23h44m23s"
time="2021-07-19T14:49:20.577Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:20.577Z" level=warning msg="Failed to acquire the lock" namespace=e2e workflow=test-webhooks-9ms6g
time="2021-07-19T14:49:20.577Z" level=info msg="Checking daemoned children of " namespace=e2e workflow=test-webhooks-9ms6g
time="2021-07-19T14:49:20.577Z" level=info msg="Queueing Error workflow gradient-perfsuite/gradient-perfsuite-lpgvn for delete in 21h57m8s"
time="2021-07-19T14:49:20.579Z" level=info msg="Queueing Failed workflow e2e/test-gantry-mvpxc for delete in 22h48m17s"
time="2021-07-19T14:49:20.580Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:20.580Z" level=warning msg="Failed to acquire the lock" namespace=e2e workflow=test-webhooks-2bwkk
time="2021-07-19T14:49:20.580Z" level=info msg="Checking daemoned children of " namespace=e2e workflow=test-webhooks-2bwkk
time="2021-07-19T14:49:20.581Z" level=info msg="Queueing Failed workflow e2e/test-gantry-8l7fz for delete in 23h29m5s"
time="2021-07-19T14:49:20.582Z" level=info msg="Queueing Failed workflow e2e/test-gantry-br7v9 for delete in 23h18m48s"
time="2021-07-19T14:49:20.583Z" level=info msg="Queueing Failed workflow e2e/test-gantry-m8k24 for delete in 23h5m51s"
time="2021-07-19T14:49:20.584Z" level=info msg="Get configmaps 200"
time="2021-07-19T14:49:20.585Z" level=warning msg="Workflow processing has been postponed due to concurrency limit" namespace=e2e workflow=webhook-test-mg7vj
time="2021-07-19T14:49:20.585Z" level=panic msg="workflow is already fulfilled" fromPhase=Failed namespace=e2e toPhase=Pending workflow=webhook-test-mg7vj
time="2021-07-19T14:49:20.585Z" level=info msg="Lock has been released by e2e/test-webhooks-5frj8. Available locks: 1" semaphore=e2e/ConfigMap/semaphore/all-workflows
time="2021-07-19T14:49:20.585Z" level=info msg="e2e/test-webhooks-5frj8 released a lock from e2e/ConfigMap/semaphore/all-workflows"
time="2021-07-19T14:49:20.585Z" level=info msg="Released all acquired locks" key=test-webhooks-5frj8 namespace=e2e workflow=test-webhooks-5frj8
time="2021-07-19T14:49:20.585Z" level=error msg="Recovered from panic" namespace=e2e r="&{0xc0000f9c70 map[fromPhase:Failed namespace:e2e toPhase:Pending workflow:webhook-test-mg7vj] 2021-07-19 14:49:20.585139239 +0000 UTC m=+0.653084461 panic <nil> workflow is already fulfilled <nil> <nil> }" stack="goroutine 226 [running]:\nruntime/debug.Stack(0xc0f0a85b82, 0x1e9f060, 0xc00042f730)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9f\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2(0xc0011aa370, 0x2198d40, 0xc000128000)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:185 +0xd1\npanic(0x1e9f060, 0xc00042f730)\n\t/usr/local/go/src/runtime/panic.go:975 +0x47a\ngithub.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020192f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345\ngithub.com/sirupsen/logrus.(*Entry).Log(0xc00042f6c0, 0x0, 0xc002168ba8, 0x1, 0x1)\n\t/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0\ngithub.com/sirupsen/logrus.(*Entry).Panic(0xc00042f6c0, 0xc002168ba8, 0x1, 0x1)\n\t/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed7572, 0x7, 0xc0013821e0, 0x49)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0xc0011aa370, 0x2198d40, 0xc000128000)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:233 +0x3ca5\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0xc000100c00, 0x2198d40, 0xc000128000, 0x0)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:754 +0x7fb\ngithub.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0xc000100c00)\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:675 +0x9b\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00146b1b0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:155 +0x5f\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00146b1b0, 0x214a840, 0xc0019ea660, 0x2020202020202001, 0xc000ca6480)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:156 +0xad\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00146b1b0, 0x3b9aca00, 0x0, 0x2020202020202001, 0xc000ca6480)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:133 +0x98\nk8s.io/apimachinery/pkg/util/wait.Until(0xc00146b1b0, 0x3b9aca00, 0xc000ca6480)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:90 +0x4d\ncreated by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).startLeading\n\t/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:298 +0x48a\n" workflow=webhook-test-mg7vj
time="2021-07-19T14:49:20.585Z" level=panic msg="workflow is already fulfilled" fromPhase=Failed namespace=e2e toPhase=Error workflow=webhook-test-mg7vj
time="2021-07-19T14:49:20.585Z" level=info msg="Checking daemoned children of " namespace=e2e workflow=webhook-test-mg7vj
time="2021-07-19T14:49:20.586Z" level=info msg="Queueing Failed workflow e2e/test-gantry-zvp4p for delete in 22h46m2s"
time="2021-07-19T14:49:20.586Z" level=info msg="Released all acquired locks" key=test-webhooks-6rrzs namespace=e2e workflow=test-webhooks-6rrzs
time="2021-07-19T14:49:20.588Z" level=info msg="Released all acquired locks" key=webhook-test-mg7vj namespace=e2e workflow=webhook-test-mg7vj
time="2021-07-19T14:49:20.590Z" level=info msg="Queueing Failed workflow e2e/test-gantry-92wcj for delete in 22h17m13s"
time="2021-07-19T14:49:20.671Z" level=info msg="List workflows 200"
time="2021-07-19T14:49:20.808Z" level=info msg="Watch workflows 200"
time="2021-07-19T14:49:20.809Z" level=info msg="Queueing Failed workflow e2e/test-gantry-92wcj for delete in 22h17m13s"
time="2021-07-19T14:49:20.810Z" level=info msg="Queueing Failed workflow e2e/test-gantry-gm72l for delete in 22h40m5s"
time="2021-07-19T14:49:20.816Z" level=info msg="Queueing Failed workflow e2e/test-gantry-lxr6d for delete in 22h25m21s"
time="2021-07-19T14:49:20.830Z" level=info msg="Queueing Succeeded workflow gradient-perfsuite/gradient-perfsuite-5xqvn for delete in 20h56m38s"
time="2021-07-19T14:49:20.834Z" level=info msg="Queueing Failed workflow e2e/test-gantry-8l7fz for delete in 23h29m5s"
time="2021-07-19T14:49:20.838Z" level=info msg="Queueing Failed workflow e2e/test-gantry-zvp4p for delete in 22h46m2s"
time="2021-07-19T14:49:20.841Z" level=info msg="Queueing Failed workflow e2e/test-gantry-br7v9 for delete in 23h18m48s"
time="2021-07-19T14:49:20.853Z" level=info msg="Queueing Error workflow gradient-perfsuite/gradient-perfsuite-492wn for delete in 22h58m57s"
time="2021-07-19T14:49:20.857Z" level=info msg="Queueing Failed workflow e2e/test-gantry-nxstx for delete in 23h10m42s"
time="2021-07-19T14:49:20.860Z" level=info msg="Queueing Failed workflow e2e/test-gantry-j4xr7 for delete in 23h44m23s"
time="2021-07-19T14:49:20.864Z" level=info msg="Queueing Failed workflow e2e/test-gantry-l64l9 for delete in 23h54m17s"
time="2021-07-19T14:49:20.871Z" level=info msg="Queueing Failed workflow e2e/test-webhooks-lv6x6 for delete in 22h17m16s"
time="2021-07-19T14:49:20.873Z" level=info msg="Queueing Succeeded workflow gradient-perfsuite/gradient-perfsuite-wvjnc for delete in 1h14m39s"
time="2021-07-19T14:49:20.877Z" level=info msg="Queueing Error workflow gradient-perfsuite/gradient-perfsuite-lpgvn for delete in 21h57m8s"
time="2021-07-19T14:49:20.882Z" level=info msg="Queueing Failed workflow e2e/test-gantry-m8k24 for delete in 23h5m51s"
time="2021-07-19T14:49:20.903Z" level=info msg="Queueing Failed workflow e2e/test-gantry-mvpxc for delete in 22h48m17s"
time="2021-07-19T14:49:22.889Z" level=info msg="Update workflows 200"
time="2021-07-19T14:49:22.906Z" level=info msg="Workflow update successful" namespace=e2e phase=Failed resourceVersion=774138743 workflow=test-webhooks-5frj8
time="2021-07-19T14:49:23.003Z" level=info msg="Update workflows 200"
time="2021-07-19T14:49:23.007Z" level=info msg="Workflow update successful" namespace=e2e phase=Failed resourceVersion=774139975 workflow=webhook-test-mg7vj
E0719 14:49:23.018352 1 runtime.go:78] Observed a panic: &logrus.Entry{Logger:(*logrus.Logger)(0xc0000f9c70), Data:logrus.Fields{"fromPhase":"Failed", "namespace":"e2e", "toPhase":"Error", "workflow":"webhook-test-mg7vj"}, Time:time.Time{wall:0xc035821c22e6024a, ext:653444363, loc:(*time.Location)(0x2e60e20)}, Level:0x0, Caller:(*runtime.Frame)(nil), Message:"workflow is already fulfilled", Buffer:(*bytes.Buffer)(nil), Context:context.Context(nil), err:""} (&{0xc0000f9c70 map[fromPhase:Failed namespace:e2e toPhase:Error workflow:webhook-test-mg7vj] 2021-07-19 14:49:20.58549921 +0000 UTC m=+0.653444363 panic <nil> workflow is already fulfilled <nil> <nil> })
goroutine 226 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1e9f060, 0xc00042f9d0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x2e35948, 0x1, 0x1)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1e9f060, 0xc00042f9d0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020193b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f960, 0x0, 0xc0021680f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f960, 0xc000b0c0f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed4558, 0x5, 0xc000b2ad00, 0xcd)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc0011aa370, 0x2198d40, 0xc000128000, 0x2147ec0, 0xc0016c0880)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1974 +0x7c
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:189 +0x32c
panic(0x1e9f060, 0xc00042f730)
/usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020192f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f6c0, 0x0, 0xc002168ba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f6c0, 0xc000b0cba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed7572, 0x7, 0xc0013821e0, 0x49)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:233 +0x3ca5
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0xc000100c00, 0x2198d40, 0xc000128000, 0x0)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:754 +0x7fb
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0xc000100c00)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:675 +0x9b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00146b1b0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00146b1b0, 0x214a840, 0xc0019ea660, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00146b1b0, 0x3b9aca00, 0x0, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00146b1b0, 0x3b9aca00, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:90 +0x4d
created by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).startLeading
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:298 +0x48a
E0719 14:49:23.018530 1 runtime.go:78] Observed a panic: &logrus.Entry{Logger:(*logrus.Logger)(0xc0000f9c70), Data:logrus.Fields{"fromPhase":"Failed", "namespace":"e2e", "toPhase":"Error", "workflow":"webhook-test-mg7vj"}, Time:time.Time{wall:0xc035821c22e6024a, ext:653444363, loc:(*time.Location)(0x2e60e20)}, Level:0x0, Caller:(*runtime.Frame)(nil), Message:"workflow is already fulfilled", Buffer:(*bytes.Buffer)(nil), Context:context.Context(nil), err:""} (&{0xc0000f9c70 map[fromPhase:Failed namespace:e2e toPhase:Error workflow:webhook-test-mg7vj] 2021-07-19 14:49:20.58549921 +0000 UTC m=+0.653444363 panic <nil> workflow is already fulfilled <nil> <nil> })
goroutine 226 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1e9f060, 0xc00042f9d0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x2e35948, 0x1, 0x1)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:51 +0xce
panic(0x1e9f060, 0xc00042f9d0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020193b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f960, 0x0, 0xc0021680f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f960, 0xc000b0c0f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed4558, 0x5, 0xc000b2ad00, 0xcd)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc0011aa370, 0x2198d40, 0xc000128000, 0x2147ec0, 0xc0016c0880)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1974 +0x7c
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:189 +0x32c
panic(0x1e9f060, 0xc00042f730)
/usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020192f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f6c0, 0x0, 0xc002168ba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f6c0, 0xc000b0cba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed7572, 0x7, 0xc0013821e0, 0x49)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:233 +0x3ca5
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0xc000100c00, 0x2198d40, 0xc000128000, 0x0)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:754 +0x7fb
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0xc000100c00)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:675 +0x9b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00146b1b0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00146b1b0, 0x214a840, 0xc0019ea660, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00146b1b0, 0x3b9aca00, 0x0, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00146b1b0, 0x3b9aca00, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:90 +0x4d
created by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).startLeading
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:298 +0x48a
E0719 14:49:23.018677 1 runtime.go:78] Observed a panic: &logrus.Entry{Logger:(*logrus.Logger)(0xc0000f9c70), Data:logrus.Fields{"fromPhase":"Failed", "namespace":"e2e", "toPhase":"Error", "workflow":"webhook-test-mg7vj"}, Time:time.Time{wall:0xc035821c22e6024a, ext:653444363, loc:(*time.Location)(0x2e60e20)}, Level:0x0, Caller:(*runtime.Frame)(nil), Message:"workflow is already fulfilled", Buffer:(*bytes.Buffer)(nil), Context:context.Context(nil), err:""} (&{0xc0000f9c70 map[fromPhase:Failed namespace:e2e toPhase:Error workflow:webhook-test-mg7vj] 2021-07-19 14:49:20.58549921 +0000 UTC m=+0.653444363 panic <nil> workflow is already fulfilled <nil> <nil> })
goroutine 226 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1e9f060, 0xc00042f9d0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1e9f060, 0xc00042f9d0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x2e35948, 0x1, 0x1)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x1e9f060, 0xc00042f9d0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020193b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f960, 0x0, 0xc0021680f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f960, 0xc000b0c0f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed4558, 0x5, 0xc000b2ad00, 0xcd)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc0011aa370, 0x2198d40, 0xc000128000, 0x2147ec0, 0xc0016c0880)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1974 +0x7c
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:189 +0x32c
panic(0x1e9f060, 0xc00042f730)
/usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020192f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f6c0, 0x0, 0xc002168ba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f6c0, 0xc000b0cba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed7572, 0x7, 0xc0013821e0, 0x49)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:233 +0x3ca5
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0xc000100c00, 0x2198d40, 0xc000128000, 0x0)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:754 +0x7fb
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0xc000100c00)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:675 +0x9b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00146b1b0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00146b1b0, 0x214a840, 0xc0019ea660, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00146b1b0, 0x3b9aca00, 0x0, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00146b1b0, 0x3b9aca00, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:90 +0x4d
created by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).startLeading
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:298 +0x48a
panic: (*logrus.Entry) 0xc00042f730 [recovered]
panic: (*logrus.Entry) 0xc00042f9d0 [recovered]
panic: (*logrus.Entry) 0xc00042f9d0 [recovered]
panic: (*logrus.Entry) 0xc00042f9d0
goroutine 226 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x1e9f060, 0xc00042f9d0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x2e35948, 0x1, 0x1)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x1e9f060, 0xc00042f9d0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020193b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f960, 0x0, 0xc0021680f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f960, 0xc000b0c0f8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed4558, 0x5, 0xc000b2ad00, 0xcd)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc0011aa370, 0x2198d40, 0xc000128000, 0x2147ec0, 0xc0016c0880)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1974 +0x7c
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func2(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:189 +0x32c
panic(0x1e9f060, 0xc00042f730)
/usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/sirupsen/logrus.Entry.log(0xc0000f9c70, 0xc0020192f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00042f6c0, 0x0, 0xc002168ba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Panic(0xc00042f6c0, 0xc000b0cba8, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:325 +0x55
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc0011aa370, 0x2198d40, 0xc000128000, 0x1ed7572, 0x7, 0xc0013821e0, 0x49)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:1855 +0x2bd
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate(0xc0011aa370, 0x2198d40, 0xc000128000)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:233 +0x3ca5
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem(0xc000100c00, 0x2198d40, 0xc000128000, 0x0)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:754 +0x7fb
github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).runWorker(0xc000100c00)
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:675 +0x9b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00146b1b0)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00146b1b0, 0x214a840, 0xc0019ea660, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00146b1b0, 0x3b9aca00, 0x0, 0x2020202020202001, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00146b1b0, 0x3b9aca00, 0xc000ca6480)
/go/pkg/mod/k8s.io/apimachinery@v0.19.6/pkg/util/wait/wait.go:90 +0x4d
created by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).startLeading
/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:298 +0x48a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment