Skip to content

Instantly share code, notes, and snippets.

@kayrus
Last active July 3, 2019 21:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kayrus/db2a74f95c587e1e475813dd0c031eb3 to your computer and use it in GitHub Desktop.
Save kayrus/db2a74f95c587e1e475813dd0c031eb3 to your computer and use it in GitHub Desktop.
shoot cluster deletion
$ kubectl --kubeconfig soil -n shoot--garden--region1 logs -f cloud-controller-manager-6f5548f8f6-wk4fg --tail 10
I0703 20:05:23.369277 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:05:30.370979 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:05:33.001121 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:05:33.077097 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:05:33.167723 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:05:33.382567 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:05:40.386265 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:05:43.014967 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:05:43.092345 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:05:43.184508 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:05:43.393297 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:05:50.397698 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:05:53.031840 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:05:53.107661 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:05:53.195670 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:05:53.407194 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:06:00.413511 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:06:03.042171 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:06:03.120906 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:06:03.207477 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:06:03.418784 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:06:10.425083 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:06:13.055954 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:06:13.143100 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:06:13.220205 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:06:13.430868 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:06:20.437329 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:06:23.071895 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:06:23.159521 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:06:23.233495 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:06:23.442315 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:06:30.450996 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:06:33.084860 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:06:33.173042 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:06:33.247589 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:06:33.456431 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:06:40.462994 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:06:43.100788 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:06:43.186523 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:06:43.257703 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:06:43.466512 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:06:50.474963 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:06:53.114980 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:06:53.201658 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:06:53.269456 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:06:53.476974 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:00.488426 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:07:02.034779 1 service_controller.go:729] Service has been deleted extension-provider-openstack-m95vp/gardener-extension-provider-openstack. Attempting to cleanup load balancer resources
I0703 20:07:03.129325 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:03.214627 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:03.283384 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:03.491197 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:10.501355 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:07:13.141579 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:13.228878 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:13.296453 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:13.502410 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:20.518094 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:07:23.162784 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:23.245682 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:23.312089 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:23.516544 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:24.337070 1 service_controller.go:729] Service has been deleted garden/alertmanager. Attempting to cleanup load balancer resources
I0703 20:07:24.337098 1 service_controller.go:729] Service has been deleted garden/alertmanager-client. Attempting to cleanup load balancer resources
I0703 20:07:24.337121 1 service_controller.go:729] Service has been deleted garden/prometheus-web. Attempting to cleanup load balancer resources
I0703 20:07:30.533520 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:07:33.181652 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:33.261617 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:33.276831 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:33.324039 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:33.381259 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:33.533947 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:35.280576 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:35.291835 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:35.325117 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:35.333388 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:35.419553 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:35.461638 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:35.475776 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:35.482417 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:35.975047 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:07:35.980352 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:07:40.551354 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zf7zq is registered without the cloud taint. Will not process.
I0703 20:07:43.193341 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-m7wp4 is registered without the cloud taint. Will not process.
I0703 20:07:43.277103 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:43.336670 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-v8gsv is registered without the cloud taint. Will not process.
I0703 20:07:43.549547 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:07:53.290035 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:07:53.567364 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-6kjl7 is registered without the cloud taint. Will not process.
I0703 20:08:03.305624 1 node_controller.go:329] This node shoot--garden--region1-cpu-worker-z1-5bb54fbdbb-zt9mc is registered without the cloud taint. Will not process.
I0703 20:08:14.388442 1 service_controller.go:729] Service has been deleted kube-system/addons-nginx-ingress-controller. Attempting to cleanup load balancer resources
I0703 20:08:14.388939 1 event.go:221] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"addons-nginx-ingress-controller", UID:"1eeac921-9d7f-11e9-b940-aa839fa8a29b", APIVersion:"v1", ResourceVersion:"7828", FieldPath:""}): type: 'Normal' reason: 'DeletingLoadBalancer' Deleting load balancer
I0703 20:08:14.427644 1 service_controller.go:729] Service has been deleted kube-system/addons-nginx-ingress-nginx-ingress-k8s-backend. Attempting to cleanup load balancer resources
I0703 20:08:14.457789 1 service_controller.go:729] Service has been deleted kube-system/kube-dns. Attempting to cleanup load balancer resources
I0703 20:08:14.516492 1 service_controller.go:729] Service has been deleted kube-system/node-exporter. Attempting to cleanup load balancer resources
I0703 20:08:14.516519 1 service_controller.go:729] Service has been deleted kube-system/kubernetes-dashboard. Attempting to cleanup load balancer resources
I0703 20:08:14.520189 1 service_controller.go:729] Service has been deleted kube-system/vpn-shoot. Attempting to cleanup load balancer resources
I0703 20:08:14.520415 1 event.go:221] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"vpn-shoot", UID:"1ef64598-9d7f-11e9-b940-aa839fa8a29b", APIVersion:"v1", ResourceVersion:"10290", FieldPath:""}): type: 'Normal' reason: 'DeletingLoadBalancer' Deleting load balancer
I0703 20:08:14.537068 1 service_controller.go:729] Service has been deleted kube-system/blackbox-exporter. Attempting to cleanup load balancer resources
I0703 20:08:14.541242 1 service_controller.go:729] Service has been deleted kube-system/calico-typha. Attempting to cleanup load balancer resources
I0703 20:08:14.552983 1 service_controller.go:729] Service has been deleted kube-system/metrics-server. Attempting to cleanup load balancer resources
E0703 20:08:19.307376 1 node_controller.go:224] Error monitoring node status: nodes is forbidden: User "system:serviceaccount:kube-system:cloud-node-controller" cannot list resource "nodes" in API group "" at the cluster scope
E0703 20:08:19.480676 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=53265, ErrCode=NO_ERROR, debug=""
E0703 20:08:19.480958 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=53265, ErrCode=NO_ERROR, debug=""
E0703 20:08:19.486697 1 reflector.go:237] k8s.io/client-go/informers/factory.go:131: Failed to watch *v1.Service: Get https://kube-apiserver/api/v1/services?resourceVersion=171229&timeoutSeconds=321&watch=true: dial tcp 198.19.84.35:443: connect: connection refused
E0703 20:08:19.486948 1 reflector.go:237] k8s.io/client-go/informers/factory.go:131: Failed to watch *v1.Node: Get https://kube-apiserver/api/v1/nodes?resourceVersion=171073&timeoutSeconds=331&watch=true: dial tcp 198.19.84.35:443: connect: connection refused
I0703 20:08:30.795921 1 leaderelection.go:249] failed to renew lease kube-system/cloud-controller-manager: failed to tryAcquireOrRenew context deadline exceeded
F0703 20:08:30.795966 1 controllermanager.go:195] leaderelection lost
rpc error: code = Unknown desc = Error: No such container: 22369e23365718c9d21af2758239a8ec7265a32ca98e756bd0f530e4d25e7ca9
time="2019-07-03T20:07:12Z" level=info msg="Starting flow" flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Syncing shoot cluster information to seed"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waking up control plane to ensure proper cleanup of resources"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deploying cloud provider account secret"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Initializing connection to Shoot"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deleting cluster autoscaler"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deleting extension resources"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deleting Shoot monitoring stack in Seed"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Cleaning up webhooks"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waiting until extension resources have been deleted"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deploying Shoot certificates / keys"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deploying Shoot control plane"
time="2019-07-03T20:07:12Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deploying Kubernetes controller manager"
time="2019-07-03T20:07:17Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waiting until shoot control plane has been reconciled"
time="2019-07-03T20:07:17Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waiting until kube-controller-manager is active"
time="2019-07-03T20:07:24Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Cleaning extended API groups"
time="2019-07-03T20:07:34Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Cleaning kubernetes resources"
time="2019-07-03T20:07:34Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Destroying Shoot workers"
time="2019-07-03T20:08:14Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waiting until shoot worker nodes have been terminated"
time="2019-07-03T20:08:14Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deleting managed resources"
time="2019-07-03T20:08:19Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waiting until managed resources have been deleted"
time="2019-07-03T20:08:19Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Destroying Kube2IAM resources"
time="2019-07-03T20:08:19Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deleting Kubernetes API server"
time="2019-07-03T20:08:19Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Destroying Shoot control plane"
time="2019-07-03T20:08:19Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Deleting backup infrastructure"
time="2019-07-03T20:08:24Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waiting until shoot control plane has been destroyed"
time="2019-07-03T20:08:24Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Destroying Shoot infrastructure"
time="2019-07-03T20:08:29Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Destroying ingress DNS record"
time="2019-07-03T20:08:34Z" level=info msg=Succeeded flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Destroying external domain DNS record"
time="2019-07-03T20:18:24Z" level=error msg=Error error="Failed to delete infrastructure: timed out waiting for the condition" flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1 task="Waiting until shoot infrastructure has been destroyed"
time="2019-07-03T20:18:24Z" level=info msg="Finished flow" flow="Shoot cluster deletion" operation=reconcile shoot=garden/region1
time="2019-07-03T20:18:24Z" level=error msg="Error deleting Shoot \"region1\": flow \"Shoot cluster deletion\" encountered task errors: [task \"Waiting until shoot infrastructure has been destroyed\" failed: Failed to delete infrastructure: timed out waiting for the condition]" operation=reconcile shoot=garden/region1
$ kubectl --kubeconfig soil -n shoot--garden--region1 logs -f -l role=apiserver -c kube-apiserver
I0703 20:04:33.989758 1 log.go:172] http: TLS handshake error from 100.130.143.192:41022: EOF
I0703 20:04:57.618613 1 log.go:172] http: TLS handshake error from 100.130.126.192:42208: EOF
I0703 20:04:58.019574 1 log.go:172] http: TLS handshake error from 100.130.126.192:51532: EOF
I0703 20:05:04.023349 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:05:04.027375 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:05:33.959667 1 log.go:172] http: TLS handshake error from 100.130.143.192:44308: EOF
I0703 20:05:57.645651 1 log.go:172] http: TLS handshake error from 100.130.126.192:45334: EOF
I0703 20:05:57.951714 1 log.go:172] http: TLS handshake error from 100.130.126.192:55824: EOF
I0703 20:04:45.976877 1 log.go:172] http: TLS handshake error from 100.130.9.192:59678: EOF
I0703 20:04:59.556057 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:04:59.565595 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:05:09.587975 1 log.go:172] http: TLS handshake error from 100.130.9.192:40076: EOF
I0703 20:05:33.567411 1 log.go:172] http: TLS handshake error from 100.130.143.192:34332: EOF
I0703 20:05:45.623892 1 log.go:172] http: TLS handshake error from 10.180.1.18:44510: EOF
I0703 20:05:59.565806 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:05:59.577931 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:06:10.005052 1 log.go:172] http: TLS handshake error from 10.180.1.18:46560: EOF
I0703 20:06:21.584609 1 log.go:172] http: TLS handshake error from 100.130.73.128:46262: EOF
I0703 20:06:04.027603 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:06:04.031101 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:04:32.836556 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:04:45.612502 1 log.go:172] http: TLS handshake error from 100.130.168.128:41536: EOF
I0703 20:05:09.964886 1 log.go:172] http: TLS handshake error from 100.130.168.128:41760: EOF
I0703 20:05:21.596674 1 log.go:172] http: TLS handshake error from 100.130.73.128:42870: EOF
I0703 20:05:21.975766 1 log.go:172] http: TLS handshake error from 100.130.73.128:46112: EOF
I0703 20:05:32.836663 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:05:32.845867 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:05:45.959073 1 log.go:172] http: TLS handshake error from 100.130.9.192:35274: EOF
I0703 20:06:09.575479 1 log.go:172] http: TLS handshake error from 100.130.9.192:43348: EOF
I0703 20:06:22.000717 1 log.go:172] http: TLS handshake error from 100.130.73.128:51370: EOF
I0703 20:06:32.846033 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:06:32.890336 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:06:33.597627 1 log.go:172] http: TLS handshake error from 100.130.143.192:37726: EOF
I0703 20:06:33.972520 1 log.go:172] http: TLS handshake error from 100.130.143.192:49526: EOF
I0703 20:06:45.572058 1 log.go:172] http: TLS handshake error from 10.180.1.18:47914: EOF
I0703 20:06:46.012555 1 log.go:172] http: TLS handshake error from 100.130.9.192:40478: EOF
I0703 20:06:57.043926 1 store.go:1414] Monitoring clusterinformations.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/clusterinformations
I0703 20:06:57.603615 1 log.go:172] http: TLS handshake error from 10.180.1.19:48800: EOF
I0703 20:06:57.924201 1 log.go:172] http: TLS handshake error from 100.130.126.192:32778: EOF
I0703 20:06:59.578109 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:06:59.588361 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:07:01.910209 1 controller.go:608] quota admission added evaluator for: { serviceaccounts}
I0703 20:07:04.031266 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:07:04.034611 1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0703 20:07:09.573354 1 log.go:172] http: TLS handshake error from 100.130.9.192:46664: EOF
I0703 20:07:09.947033 1 log.go:172] http: TLS handshake error from 100.130.168.128:51224: EOF
E0703 20:07:17.486994 1 crd_finalizer.go:275] azuremachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "azuremachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.485901 1 crd_finalizer.go:275] alicloudmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "alicloudmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.484393 1 crd_finalizer.go:275] awsmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "awsmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.493094 1 crd_finalizer.go:275] azuremachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "azuremachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.496643 1 crd_finalizer.go:275] alicloudmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "alicloudmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:17.496565 1 cacher.go:606] cacher (*apiextensions.CustomResourceDefinition): 1 objects queued in incoming channel.
E0703 20:07:17.498546 1 crd_finalizer.go:275] dnsentries.dns.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "dnsentries.dns.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.496689 1 crd_finalizer.go:275] awsmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "awsmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.503597 1 crd_finalizer.go:275] dnsowners.dns.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "dnsowners.dns.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.503467 1 crd_finalizer.go:275] azuremachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "azuremachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.513285 1 crd_finalizer.go:275] controlplanes.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "controlplanes.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:17.504925 1 store.go:1414] Monitoring awsmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/awsmachineclasses
I0703 20:07:17.512456 1 cacher.go:606] cacher (*apiextensions.CustomResourceDefinition): 1 objects queued in incoming channel.
I0703 20:07:17.512496 1 cacher.go:606] cacher (*apiextensions.CustomResourceDefinition): 2 objects queued in incoming channel.
I0703 20:07:17.512514 1 cacher.go:606] cacher (*apiextensions.CustomResourceDefinition): 3 objects queued in incoming channel.
E0703 20:07:17.524289 1 crd_finalizer.go:275] dnsowners.dns.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "dnsowners.dns.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.524641 1 crd_finalizer.go:275] dnsentries.dns.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "dnsentries.dns.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:17.533271 1 store.go:1414] Monitoring azuremachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/azuremachineclasses
I0703 20:07:17.542093 1 store.go:1414] Monitoring alicloudmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/alicloudmachineclasses
I0703 20:07:17.539595 1 cacher.go:606] cacher (*apiextensions.CustomResourceDefinition): 1 objects queued in incoming channel.
E0703 20:07:17.547946 1 crd_finalizer.go:275] controlplanes.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "controlplanes.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.555339 1 crd_finalizer.go:275] awsmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "awsmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
W0703 20:07:17.556437 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:17.566780 1 crd_finalizer.go:275] alicloudmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "alicloudmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
W0703 20:07:17.573559 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:07:17.579896 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
I0703 20:07:17.577602 1 store.go:1414] Monitoring azuremachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/azuremachineclasses
I0703 20:07:17.574487 1 cacher.go:606] cacher (*apiextensions.CustomResourceDefinition): 2 objects queued in incoming channel.
W0703 20:07:17.583179 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
I0703 20:07:17.575766 1 store.go:1414] Monitoring alicloudmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/alicloudmachineclasses
E0703 20:07:17.582716 1 crd_finalizer.go:275] azuremachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "azuremachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/azuremachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f97719-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:17.593808 1 crd_finalizer.go:275] clusters.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "clusters.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.593856 1 crd_finalizer.go:275] extensions.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "extensions.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
W0703 20:07:17.584111 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:07:17.589869 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:07:17.591142 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:17.593884 1 crd_finalizer.go:275] dnsproviders.dns.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "dnsproviders.dns.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.599986 1 crd_finalizer.go:275] alicloudmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "alicloudmachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/alicloudmachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f414cb-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:17.600131 1 crd_finalizer.go:275] azuremachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "azuremachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/azuremachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f97719-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:17.601807 1 crd_finalizer.go:275] clusters.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "clusters.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:17.603039 1 store.go:1414] Monitoring gcpmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/gcpmachineclasses
E0703 20:07:17.617064 1 crd_finalizer.go:131] Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "gcpmachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/gcpmachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1fae76e-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:17.617086 1 crd_finalizer.go:275] gcpmachineclasses.machine.sapcloud.io failed with: grpc: the client connection is closing
W0703 20:07:17.610828 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:07:17.614088 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:07:17.617493 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:17.622578 1 autoregister_controller.go:190] v1alpha1.dns.gardener.cloud failed with : apiservices.apiregistration.k8s.io "v1alpha1.dns.gardener.cloud" not found
I0703 20:07:17.614704 1 store.go:1414] Monitoring gcpmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/gcpmachineclasses
E0703 20:07:17.625450 1 crd_finalizer.go:275] gcpmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "gcpmachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/gcpmachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1fae76e-9d87-11e9-810c-5675402bf49a, UID in object meta:
I0703 20:07:17.634144 1 store.go:1414] Monitoring dnsproviders.dns.gardener.cloud count at <storage-prefix>//dns.gardener.cloud/dnsproviders
E0703 20:07:17.643242 1 crd_finalizer.go:275] dnsproviders.dns.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "dnsproviders.dns.gardener.cloud": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/dnsproviders.dns.gardener.cloud, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: be6a5895-9d87-11e9-b940-aa839fa8a29b, UID in object meta:
E0703 20:07:17.621351 1 autoregister_controller.go:190] v1alpha1.dns.gardener.cloud failed with : apiservices.apiregistration.k8s.io "v1alpha1.dns.gardener.cloud" not found
E0703 20:07:17.683633 1 crd_finalizer.go:275] infrastructures.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "infrastructures.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:17.650980 1 store.go:1414] Monitoring extensions.extensions.gardener.cloud count at <storage-prefix>//extensions.gardener.cloud/extensions
E0703 20:07:17.656340 1 crd_finalizer.go:131] Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "extensions.extensions.gardener.cloud": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/extensions.extensions.gardener.cloud, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: bea76157-9d87-11e9-b940-aa839fa8a29b, UID in object meta:
E0703 20:07:17.656364 1 crd_finalizer.go:275] extensions.extensions.gardener.cloud failed with: context canceled
E0703 20:07:17.678673 1 crd_finalizer.go:275] infrastructures.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "infrastructures.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
W0703 20:07:17.693594 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:17.877897 1 crd_finalizer.go:275] machinedeployments.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machinedeployments.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:17.889311 1 crd_finalizer.go:275] machinedeployments.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machinedeployments.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:17.893184 1 store.go:1414] Monitoring machinedeployments.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/machinedeployments
E0703 20:07:17.908692 1 crd_finalizer.go:275] machinedeployments.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machinedeployments.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/machinedeployments.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f03040-9d87-11e9-810c-5675402bf49a, UID in object meta:
W0703 20:07:17.909278 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
I0703 20:07:17.911873 1 store.go:1414] Monitoring machinedeployments.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/machinedeployments
E0703 20:07:17.916709 1 crd_finalizer.go:131] Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machinedeployments.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/machinedeployments.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f03040-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:17.916729 1 crd_finalizer.go:275] machinedeployments.machine.sapcloud.io failed with: context canceled
E0703 20:07:18.077842 1 crd_finalizer.go:275] machines.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machines.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:18.083789 1 crd_finalizer.go:275] machines.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machines.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:18.100119 1 store.go:1414] Monitoring machines.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/machines
W0703 20:07:18.107582 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:18.109114 1 crd_finalizer.go:275] machines.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machines.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/machines.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f2f059-9d87-11e9-810c-5675402bf49a, UID in object meta:
I0703 20:07:18.111171 1 store.go:1414] Monitoring machines.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/machines
E0703 20:07:18.115123 1 crd_finalizer.go:131] Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machines.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/machines.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f2f059-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:18.115150 1 crd_finalizer.go:275] machines.machine.sapcloud.io failed with: grpc: the client connection is closing
E0703 20:07:18.275972 1 crd_finalizer.go:275] machinesets.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machinesets.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:18.285459 1 crd_finalizer.go:275] machinesets.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machinesets.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
W0703 20:07:18.296180 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
I0703 20:07:18.292840 1 store.go:1414] Monitoring machinesets.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/machinesets
E0703 20:07:18.298858 1 crd_finalizer.go:131] Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "machinesets.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/machinesets.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1f1a0da-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:18.298941 1 crd_finalizer.go:275] machinesets.machine.sapcloud.io failed with: context canceled
E0703 20:07:18.480202 1 crd_finalizer.go:275] managedresources.resources.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "managedresources.resources.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:18.481256 1 crd_finalizer.go:275] managedresources.resources.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "managedresources.resources.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:18.490613 1 store.go:1414] Monitoring managedresources.resources.gardener.cloud count at <storage-prefix>//resources.gardener.cloud/managedresources
I0703 20:07:18.500513 1 store.go:1414] Monitoring managedresources.resources.gardener.cloud count at <storage-prefix>//resources.gardener.cloud/managedresources
E0703 20:07:18.506762 1 crd_finalizer.go:275] managedresources.resources.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "managedresources.resources.gardener.cloud": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/managedresources.resources.gardener.cloud, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: bf21711b-9d87-11e9-b940-aa839fa8a29b, UID in object meta:
W0703 20:07:18.504965 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:18.513405 1 autoregister_controller.go:190] v1alpha1.resources.gardener.cloud failed with : apiservices.apiregistration.k8s.io "v1alpha1.resources.gardener.cloud" not found
E0703 20:07:18.498288 1 crd_finalizer.go:275] managedresources.resources.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "managedresources.resources.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:18.512842 1 autoregister_controller.go:190] v1alpha1.resources.gardener.cloud failed with : apiservices.apiregistration.k8s.io "v1alpha1.resources.gardener.cloud" not found
E0703 20:07:18.676710 1 crd_finalizer.go:275] openstackmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "openstackmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:18.687053 1 crd_finalizer.go:275] openstackmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "openstackmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:18.692483 1 store.go:1414] Monitoring openstackmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/openstackmachineclasses
E0703 20:07:18.702127 1 crd_finalizer.go:275] openstackmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "openstackmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:18.699049 1 store.go:1414] Monitoring openstackmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/openstackmachineclasses
W0703 20:07:18.708608 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:18.706885 1 crd_finalizer.go:275] openstackmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "openstackmachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/openstackmachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1fc371c-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:18.885765 1 crd_finalizer.go:275] operatingsystemconfigs.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "operatingsystemconfigs.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:18.887584 1 crd_finalizer.go:275] operatingsystemconfigs.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "operatingsystemconfigs.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:18.893084 1 store.go:1414] Monitoring operatingsystemconfigs.extensions.gardener.cloud count at <storage-prefix>//extensions.gardener.cloud/operatingsystemconfigs
E0703 20:07:18.902647 1 crd_finalizer.go:275] operatingsystemconfigs.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "operatingsystemconfigs.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:18.910009 1 store.go:1414] Monitoring operatingsystemconfigs.extensions.gardener.cloud count at <storage-prefix>//extensions.gardener.cloud/operatingsystemconfigs
W0703 20:07:18.914490 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:18.915790 1 crd_finalizer.go:131] Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "operatingsystemconfigs.extensions.gardener.cloud": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/operatingsystemconfigs.extensions.gardener.cloud, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: bf5e897c-9d87-11e9-b940-aa839fa8a29b, UID in object meta:
E0703 20:07:18.915813 1 crd_finalizer.go:275] operatingsystemconfigs.extensions.gardener.cloud failed with: context canceled
E0703 20:07:19.076902 1 crd_finalizer.go:275] packetmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "packetmachineclasses.machine.sapcloud.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:19.096024 1 store.go:1414] Monitoring packetmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/packetmachineclasses
I0703 20:07:19.099247 1 store.go:1414] Monitoring packetmachineclasses.machine.sapcloud.io count at <storage-prefix>//machine.sapcloud.io/packetmachineclasses
W0703 20:07:19.108580 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:19.105913 1 crd_finalizer.go:131] Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "packetmachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/packetmachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1fd9faf-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:19.105936 1 crd_finalizer.go:275] packetmachineclasses.machine.sapcloud.io failed with: context canceled
E0703 20:07:19.113064 1 autoregister_controller.go:190] v1alpha1.machine.sapcloud.io failed with : apiservices.apiregistration.k8s.io "v1alpha1.machine.sapcloud.io" not found
E0703 20:07:19.111748 1 crd_finalizer.go:275] packetmachineclasses.machine.sapcloud.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "packetmachineclasses.machine.sapcloud.io": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/packetmachineclasses.machine.sapcloud.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c1fd9faf-9d87-11e9-810c-5675402bf49a, UID in object meta:
E0703 20:07:19.115992 1 autoregister_controller.go:190] v1alpha1.machine.sapcloud.io failed with : apiservices.apiregistration.k8s.io "v1alpha1.machine.sapcloud.io" not found
E0703 20:07:19.288280 1 crd_finalizer.go:275] workers.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "workers.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:19.292290 1 crd_finalizer.go:275] workers.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "workers.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:19.296250 1 store.go:1414] Monitoring workers.extensions.gardener.cloud count at <storage-prefix>//extensions.gardener.cloud/workers
E0703 20:07:19.304035 1 crd_finalizer.go:275] workers.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "workers.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:19.321236 1 autoregister_controller.go:190] v1alpha1.extensions.gardener.cloud failed with : apiservices.apiregistration.k8s.io "v1alpha1.extensions.gardener.cloud" not found
E0703 20:07:19.310983 1 crd_finalizer.go:275] workers.extensions.gardener.cloud failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "workers.extensions.gardener.cloud": the object has been modified; please apply your changes to the latest version and try again
W0703 20:07:19.316909 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:07:19.322006 1 autoregister_controller.go:190] v1alpha1.extensions.gardener.cloud failed with : apiservices.apiregistration.k8s.io "v1alpha1.extensions.gardener.cloud" not found
I0703 20:07:21.578254 1 log.go:172] http: TLS handshake error from 100.130.73.128:49438: EOF
I0703 20:07:21.987343 1 log.go:172] http: TLS handshake error from 100.130.73.128:55568: EOF
I0703 20:07:24.356122 1 cacher.go:606] cacher (*core.Endpoints): 1 objects queued in incoming channel.
E0703 20:07:30.643662 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0703 20:07:30.687522 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:32.890505 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:07:33.638631 1 log.go:172] http: TLS handshake error from 100.130.143.192:40750: EOF
I0703 20:07:33.926089 1 log.go:172] http: TLS handshake error from 100.130.143.192:53180: EOF
E0703 20:07:33.945187 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0703 20:07:33.945223 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0703 20:07:34.752983 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:07:34.945864 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:07:34.955394 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0703 20:07:35.483586 1 cacher.go:606] cacher (*core.Pod): 1 objects queued in incoming channel.
I0703 20:07:35.490640 1 cacher.go:606] cacher (*core.Pod): 1 objects queued in incoming channel.
I0703 20:07:35.490674 1 cacher.go:606] cacher (*core.Pod): 2 objects queued in incoming channel.
I0703 20:07:35.505350 1 cacher.go:606] cacher (*core.Pod): 1 objects queued in incoming channel.
E0703 20:07:35.527878 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:35.550343 1 cacher.go:606] cacher (*extensions.Deployment): 2 objects queued in incoming channel.
E0703 20:07:35.555755 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0703 20:07:35.599824 1 cacher.go:606] cacher (*extensions.Deployment): 3 objects queued in incoming channel.
E0703 20:07:35.620301 1 handler_proxy.go:115] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
E0703 20:07:35.622710 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0703 20:07:35.957871 1 handler_proxy.go:115] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
E0703 20:07:35.961587 1 handler_proxy.go:115] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
E0703 20:07:36.252171 1 handler_proxy.go:115] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
E0703 20:07:36.902454 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0703 20:07:36.902477 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0703 20:07:37.217215 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0703 20:07:37.217240 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0703 20:07:37.501773 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0703 20:07:37.501877 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0703 20:07:38.501823 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0703 20:07:40.822945 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0703 20:07:40.822967 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0703 20:07:41.077041 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0703 20:07:41.077313 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0703 20:07:41.077331 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0703 20:07:41.502726 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0703 20:07:41.502868 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0703 20:07:41.502909 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0703 20:07:41.822940 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0703 20:07:41.823044 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0703 20:07:41.823058 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0703 20:07:42.022906 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0703 20:07:43.415013 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0703 20:07:45.652668 1 log.go:172] http: TLS handshake error from 10.180.1.18:50790: EOF
I0703 20:07:45.973870 1 log.go:172] http: TLS handshake error from 100.130.9.192:43700: EOF
I0703 20:07:57.569181 1 log.go:172] http: TLS handshake error from 10.180.1.19:51466: EOF
I0703 20:07:57.935658 1 log.go:172] http: TLS handshake error from 10.180.1.19:35552: EOF
E0703 20:08:05.639068 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0703 20:08:09.568979 1 log.go:172] http: TLS handshake error from 100.130.9.192:49332: EOF
I0703 20:08:09.937051 1 log.go:172] http: TLS handshake error from 100.130.168.128:53998: EOF
E0703 20:08:12.039699 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0703 20:08:13.430435 1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0703 20:08:14.430927 1 crd_finalizer.go:275] globalnetworkpolicies.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.437534 1 crd_finalizer.go:275] globalnetworkpolicies.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.457431 1 cacher.go:606] cacher (*rbac.RoleBinding): 1 objects queued in incoming channel.
I0703 20:08:14.460010 1 store.go:1414] Monitoring globalnetworkpolicies.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/globalnetworkpolicies
I0703 20:08:14.477740 1 cacher.go:606] cacher (*policy.PodSecurityPolicy): 1 objects queued in incoming channel.
I0703 20:08:14.482735 1 cacher.go:606] cacher (*policy.PodSecurityPolicy): 1 objects queued in incoming channel.
I0703 20:08:14.484767 1 cacher.go:606] cacher (*extensions.DaemonSet): 1 objects queued in incoming channel.
I0703 20:08:14.483516 1 store.go:1414] Monitoring globalnetworksets.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/globalnetworksets
E0703 20:08:14.488043 1 crd_finalizer.go:275] globalnetworksets.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "globalnetworksets.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.492862 1 cacher.go:606] cacher (*extensions.DaemonSet): 1 objects queued in incoming channel.
E0703 20:08:14.493121 1 crd_finalizer.go:275] globalnetworksets.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "globalnetworksets.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.498596 1 cacher.go:606] cacher (*extensions.DaemonSet): 1 objects queued in incoming channel.
I0703 20:08:14.513838 1 cacher.go:606] cacher (*rbac.ClusterRole): 1 objects queued in incoming channel.
I0703 20:08:14.514193 1 cacher.go:606] cacher (*rbac.ClusterRole): 1 objects queued in incoming channel.
E0703 20:08:14.515781 1 crd_finalizer.go:275] networksets.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "networksets.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.523532 1 cacher.go:606] cacher (*core.Service): 1 objects queued in incoming channel.
E0703 20:08:14.504042 1 crd_finalizer.go:275] ipamconfigs.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamconfigs.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.507866 1 cacher.go:606] cacher (*rbac.ClusterRole): 1 objects queued in incoming channel.
I0703 20:08:14.507888 1 cacher.go:606] cacher (*rbac.ClusterRole): 2 objects queued in incoming channel.
E0703 20:08:14.509312 1 crd_finalizer.go:275] ipamblocks.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.509731 1 crd_finalizer.go:275] networksets.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "networksets.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.548783 1 crd_finalizer.go:275] ippools.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ippools.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.526972 1 crd_finalizer.go:275] ipamblocks.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.527190 1 crd_finalizer.go:275] ipamconfigs.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamconfigs.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.552604 1 crd_finalizer.go:275] ippools.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ippools.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.545481 1 store.go:1414] Monitoring ipamconfigs.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/ipamconfigs
E0703 20:08:14.575390 1 crd_finalizer.go:275] networkpolicies.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.564575 1 cacher.go:606] cacher (*rbac.ClusterRoleBinding): 1 objects queued in incoming channel.
E0703 20:08:14.569987 1 crd_finalizer.go:275] hostendpoints.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.570240 1 crd_finalizer.go:275] bgppeers.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "bgppeers.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.574376 1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": StorageError: invalid object, Code: 4, Key: /registry/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 300a5d61-9dce-11e9-b44c-3601b7927767, UID in object meta:
E0703 20:08:14.594954 1 crd_finalizer.go:275] globalnetworkpolicies.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.595416 1 crd_finalizer.go:275] globalnetworkpolicies.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.623024 1 crd_finalizer.go:275] hostendpoints.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.624854 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.629980 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.627553 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
I0703 20:08:14.596900 1 store.go:1414] Monitoring hostendpoints.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/hostendpoints
I0703 20:08:14.633324 1 store.go:1414] Monitoring ipamhandles.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/ipamhandles
W0703 20:08:14.638227 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.649153 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.653575 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.655908 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.647441 1 crd_finalizer.go:275] networkpolicies.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.657700 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.642930 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.645540 1 crd_finalizer.go:275] hostendpoints.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.669731 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.677341 1 crd_finalizer.go:275] ippools.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ippools.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
I0703 20:08:14.687490 1 store.go:1414] Monitoring ipamhandles.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/ipamhandles
W0703 20:08:14.690814 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
I0703 20:08:14.681795 1 store.go:1414] Monitoring felixconfigurations.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/felixconfigurations
E0703 20:08:14.693555 1 crd_finalizer.go:275] clusterinformations.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.694960 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
I0703 20:08:14.713151 1 store.go:1414] Monitoring ipamblocks.crd.projectcalico.org count at <storage-prefix>//crd.projectcalico.org/ipamblocks
W0703 20:08:14.713604 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.708810 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.735000 1 crd_finalizer.go:275] clusterinformations.crd.projectcalico.org failed with: customresourcedefinitions.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" not found
W0703 20:08:14.714016 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.718625 1 crd_finalizer.go:275] clusterinformations.crd.projectcalico.org failed with: customresourcedefinitions.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" not found
E0703 20:08:14.743766 1 crd_finalizer.go:275] felixconfigurations.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.751634 1 crd_finalizer.go:275] ipamhandles.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamhandles.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.755321 1 crd_finalizer.go:275] ipamblocks.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.760538 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.756966 1 crd_finalizer.go:275] blockaffinities.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "blockaffinities.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.757154 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.763834 1 crd_finalizer.go:275] felixconfigurations.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.764007 1 crd_finalizer.go:275] blockaffinities.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "blockaffinities.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.766503 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.770101 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.772409 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.774797 1 crd_finalizer.go:275] hostendpoints.crd.projectcalico.org failed with: customresourcedefinitions.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org" not found
E0703 20:08:14.764141 1 crd_finalizer.go:275] ipamblocks.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
W0703 20:08:14.765958 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.769412 1 crd_finalizer.go:275] ipamhandles.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "ipamhandles.crd.projectcalico.org": the object has been modified; please apply your changes to the latest version and try again
E0703 20:08:14.762163 1 crd_finalizer.go:275] ippools.crd.projectcalico.org failed with: customresourcedefinitions.apiextensions.k8s.io "ippools.crd.projectcalico.org" not found
E0703 20:08:14.774839 1 crd_finalizer.go:275] ippools.crd.projectcalico.org failed with: customresourcedefinitions.apiextensions.k8s.io "ippools.crd.projectcalico.org" not found
E0703 20:08:14.779142 1 crd_finalizer.go:275] hostendpoints.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/hostendpoints.crd.projectcalico.org, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1f4f7bc8-9d7f-11e9-b940-aa839fa8a29b, UID in object meta:
W0703 20:08:14.779997 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.780529 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.781240 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.765681 1 crd_finalizer.go:275] hostendpoints.crd.projectcalico.org failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org": StorageError: invalid object, Code: 4, Key: /registry/apiextensions.k8s.io/customresourcedefinitions/hostendpoints.crd.projectcalico.org, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1f4f7bc8-9d7f-11e9-b940-aa839fa8a29b, UID in object meta:
W0703 20:08:14.786695 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.794023 1 autoregister_controller.go:190] v1.crd.projectcalico.org failed with : apiservices.apiregistration.k8s.io "v1.crd.projectcalico.org" not found
W0703 20:08:14.782599 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
W0703 20:08:14.793981 1 cacher.go:125] Terminating all watchers from cacher *unstructured.Unstructured
E0703 20:08:14.798895 1 autoregister_controller.go:190] v1.crd.projectcalico.org failed with : apiservices.apiregistration.k8s.io "v1.crd.projectcalico.org" not found
I0703 20:08:19.450710 1 controller.go:171] Shutting down kubernetes service endpoint reconciler
I0703 20:08:19.450985 1 crdregistration_controller.go:143] Shutting down crd-autoregister controller
I0703 20:08:19.451032 1 naming_controller.go:295] Shutting down NamingConditionController
I0703 20:08:19.451045 1 customresource_discovery_controller.go:210] Shutting down DiscoveryController
I0703 20:08:19.451056 1 crd_finalizer.go:254] Shutting down CRDFinalizer
I0703 20:08:19.451067 1 available_controller.go:290] Shutting down AvailableConditionController
I0703 20:08:19.451087 1 establishing_controller.go:84] Shutting down EstablishingController
I0703 20:08:19.451101 1 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0703 20:08:19.451113 1 autoregister_controller.go:160] Shutting down autoregister controller
I0703 20:08:19.451999 1 controller.go:90] Shutting down OpenAPI AggregationController
I0703 20:08:19.452115 1 secure_serving.go:156] Stopped listening on [::]:443
E0703 20:08:19.452269 1 controller.go:173] StorageError: key not found, Code: 1, Key: /registry/masterleases/10.0.1.173, ResourceVersion: 0, AdditionalErrorMsg:
I0703 20:08:19.452236 1 controller.go:171] Shutting down kubernetes service endpoint reconciler
I0703 20:08:19.453167 1 crd_finalizer.go:254] Shutting down CRDFinalizer
I0703 20:08:19.453188 1 available_controller.go:290] Shutting down AvailableConditionController
I0703 20:08:19.453201 1 establishing_controller.go:84] Shutting down EstablishingController
I0703 20:08:19.453213 1 customresource_discovery_controller.go:210] Shutting down DiscoveryController
I0703 20:08:19.453225 1 naming_controller.go:295] Shutting down NamingConditionController
I0703 20:08:19.453248 1 crdregistration_controller.go:143] Shutting down crd-autoregister controller
I0703 20:08:19.453268 1 autoregister_controller.go:160] Shutting down autoregister controller
I0703 20:08:19.453282 1 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0703 20:08:19.453567 1 secure_serving.go:156] Stopped listening on [::]:443
I0703 20:08:19.453578 1 controller.go:90] Shutting down OpenAPI AggregationController
E0703 20:08:19.456165 1 controller.go:173] no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
I0703 20:08:19.467627 1 controller.go:171] Shutting down kubernetes service endpoint reconciler
I0703 20:08:19.468009 1 establishing_controller.go:84] Shutting down EstablishingController
I0703 20:08:19.468035 1 naming_controller.go:295] Shutting down NamingConditionController
I0703 20:08:19.468071 1 customresource_discovery_controller.go:210] Shutting down DiscoveryController
I0703 20:08:19.468095 1 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0703 20:08:19.468263 1 crdregistration_controller.go:143] Shutting down crd-autoregister controller
I0703 20:08:19.468288 1 autoregister_controller.go:160] Shutting down autoregister controller
I0703 20:08:19.468318 1 crd_finalizer.go:254] Shutting down CRDFinalizer
I0703 20:08:19.468343 1 available_controller.go:290] Shutting down AvailableConditionController
I0703 20:08:19.469473 1 secure_serving.go:156] Stopped listening on [::]:443
I0703 20:08:19.469528 1 controller.go:90] Shutting down OpenAPI AggregationController
E0703 20:08:19.470465 1 controller.go:173] StorageError: key not found, Code: 1, Key: /registry/masterleases/10.0.1.173, ResourceVersion: 0, AdditionalErrorMsg:
rpc error: code = Unknown desc = Error: No such container: 726940b2872f26f79a61ce7c700e9899fc6af1802d3866b49166e54438524bearpc error: code = Unknown desc = Error: No such container: 947ac11dad49c3036b0a77275b3bdf60e60521cca98d67b16bd86ac4ae39ce26rpc error: code = Unknown desc = Error: No such container: 3724eacc66ddbb926b0d0ce6e4461e8541ab7813001def069b882117b004f795
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment