Skip to content

Instantly share code, notes, and snippets.

@MeirP-3

MeirP-3/gist:11e8c32408c10ab9c0b29109d27f31df

Created November 4, 2020 15:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save MeirP-3/11e8c32408c10ab9c0b29109d27f31df to your computer and use it in GitHub Desktop.
Save MeirP-3/11e8c32408c10ab9c0b29109d27f31df to your computer and use it in GitHub Desktop.
Download ZIP
before log
Raw
gistfile1.txt
I1104 15:53:42.480826 12397 featureflag.go:154] FeatureFlag "Spotinst"=true
I1104 15:53:42.523945 12397 loader.go:375] Config loaded from file: /home/meirp/.kube/config
Using cluster from kubectl context: test.example.com
I1104 15:53:42.524109 12397 factory.go:68] state store s3://some-bucket/kops
I1104 15:53:42.524169 12397 s3context.go:328] unable to read /sys/devices/virtual/dmi/id/product_uuid, assuming not running on EC2: open /sys/devices/virtual/dmi/id/product_uuid: permission denied
I1104 15:53:43.947423 12397 s3context.go:163] unable to get region from metadata:unable to get region from metadata: EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/dynamic/instance-identity/document: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1104 15:53:43.947489 12397 s3context.go:173] defaulting region to "us-east-1"
I1104 15:53:45.425598 12397 s3context.go:213] found bucket in region "us-east-1"
I1104 15:53:45.425669 12397 s3fs.go:290] Reading file "s3://some-bucket/kops/test.example.com/config"
I1104 15:53:46.224185 12397 loader.go:375] Config loaded from file: /home/meirp/.kube/config
I1104 15:53:46.226240 12397 round_trippers.go:423] curl -k -v -XGET -H "Authorization: Basic **************" -H "Accept: application/json, */*" -H "User-Agent: kops/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://api.test.example.com/api/v1/nodes'
I1104 15:53:47.201973 12397 round_trippers.go:443] GET https://api.test.example.com/api/v1/nodes 200 OK in 975 milliseconds
I1104 15:53:47.202048 12397 round_trippers.go:449] Response Headers:
I1104 15:53:47.202072 12397 round_trippers.go:452] Audit-Id: *********************
I1104 15:53:47.202088 12397 round_trippers.go:452] Content-Type: application/json
I1104 15:53:47.202098 12397 round_trippers.go:452] Date: Wed, 04 Nov 2020 13:53:47 GMT
I1104 15:53:47.579059 12397 request.go:1068] Response Body: {"kind":"NodeList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/nodes","resourceVersion":"45488858"},"items":[{"metadata":{"name":"ip-10-**-**-169.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-169.ec2.internal","uid":"507b5bdd-1ea0-11eb-9aff-16da7cad7b07","resourceVersion":"45488831","creationTimestamp":"2020-11-04T13:19:05Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"t3a.2xlarge","beta.kubernetes.io/os":"linux","dedicated":"another-group","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"another-group","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-169.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":"","spotinst.io/node-lifecycle":"spot"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.115.0/24","providerID":"aws:///us-east-1f/i-0c8dfd269d1bffd81","taints":[{"key":"dedicated","value":"another-group","effect":"NoSchedule"}]},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"8","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"32665868Ki","pods":"58"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"8","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"32563468Ki","pods":"58"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:37Z","lastTransitionTime":"2020-11-04T13:19:05Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:37Z","lastTransitionTime":"2020-11-04T13:19:05Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:37Z","lastTransitionTime":"2020-11-04T13:19:05Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:53:37Z","lastTransitionTime":"2020-11-04T13:19:55Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.41.169"},{"type":"InternalDNS","address":"ip-10-**-**-169.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-169.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec2fd7328ad5220ca24cde06a441ae5a","systemUUID":"EC2FD732-8AD5-220C-A24C-DE06A441AE5A","bootID":"d8912c50-c519-4500-a536-1f43b0077e2b","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.dev.example.com/cybervision/invu-another-grouprithm-service@sha256:ec66013ab9c9e14369cd38501e09a399595a52f4b7424d9883d40009c12f705a","registry.dev.example.com/cybervision/invu-another-grouprithm-service:2.0.0-38065"],"sizeBytes":709056632},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}]}},{"metadata":{"name":"ip-10-**-**-110.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-110.ec2.internal","uid":"3d99698f-1e9c-11eb-9aff-16da7cad7b07","resourceVersion":"45488810","creationTimestamp":"2020-11-04T12:49:56Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"m5a.xlarge","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"nodes","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-110.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":"","spotinst.io/node-lifecycle":"spot"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.114.0/24","providerID":"aws:///us-east-1f/i-0b738537410c47ce9"},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16038900Ki","pods":"58"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"15936500Ki","pods":"58"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:28Z","lastTransitionTime":"2020-11-04T12:49:56Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:28Z","lastTransitionTime":"2020-11-04T12:49:56Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:28Z","lastTransitionTime":"2020-11-04T12:49:56Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:53:28Z","lastTransitionTime":"2020-11-04T12:50:16Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.42.110"},{"type":"InternalDNS","address":"ip-10-**-**-110.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-110.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec2b9b820e177da80fbd1e4c85bbf31a","systemUUID":"EC2B9B82-0E17-7DA8-0FBD-1E4C85BBF31A","bootID":"ffffbb68-1815-42b8-bf09-86fce9b807c5","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.dev.example.com/cybervision/pregsense-monitoring-session-service@sha256:eedc23e0c6848e90c6725ea2e0f821a9911a8651cdf964b48a818526c6154d99","registry.dev.example.com/cybervision/pregsense-monitoring-session-service:2.0.0-37729"],"sizeBytes":916680232},{"names":["confluentinc/cp-ksql-server@sha256:a0b75d2f9926ac31139b2bebff3ac95a24e9f08465176aef16ed3c481b39a297","confluentinc/cp-ksql-server:5.2.2"],"sizeBytes":524657779},{"names":["registry.dev.example.com/cybervision/pregsense-mail-service@sha256:633a2b3faf69477dfd745b83def6689077641309c1092121edb215bcfec06d8d","registry.dev.example.com/cybervision/pregsense-mail-service:1.1.0-38167"],"sizeBytes":331982846},{"names":["registry.dev.example.com/cybervision/pregsense-geo-service@sha256:9e44ea04116b0dad47c7774c04666e6a74de49e252ebd2b3f3d562ce63bf9bbc","registry.dev.example.com/cybervision/pregsense-geo-service:2.0.0-37561"],"sizeBytes":298502628},{"names":["example/jmx-prometheus-httpserver@sha256:725e8a9bfec05f432b931edc11b1ee8bac3895e408bb91af58266dc88ee4e63b","example/jmx-prometheus-httpserver:0.11.0"],"sizeBytes":273954021},{"names":["registry.dev.example.com/ce/middleware/kafka@sha256:5a13ea153f26ac29e73fd3a0ed67c701f33ea761a0b10a03870fdda51e3975cc","registry.dev.example.com/ce/middleware/kafka:0.6.33"],"sizeBytes":269199283},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["registry.dev.example.com/cybervision/invu-web/pro@sha256:4a75e40bc1774e59e805ff68e713542d6c839e54e550b21688f9b23ec401de28","registry.dev.example.com/cybervision/invu-web/pro:2.0.0-38041"],"sizeBytes":107688818},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["registry.dev.example.com/ce/utilities/mysql-client@sha256:a748d356fc0b118b7a2cec5839fc73d473cfc9f8294c6ac8e07064c310e22ec9","registry.dev.example.com/ce/utilities/mysql-client:latest"],"sizeBytes":38388577},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["registry.dev.example.com/ce/utilities/kt@sha256:a52550f941a9449ea205201f2c42d0352da5f70bf281d1d638944352781851b2","registry.dev.example.com/ce/utilities/kt:latest"],"sizeBytes":22871156},{"names":["busybox@sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d","busybox:latest"],"sizeBytes":1231733},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}],"volumesInUse":["kubernetes.io/aws-ebs/aws://us-east-1f/vol-04eda9d82d6dabadd"],"volumesAttached":[{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-04eda9d82d6dabadd","devicePath":"/dev/xvdbn"}]}},{"metadata":{"name":"ip-10-**-**-241.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-241.ec2.internal","uid":"33bf1cf2-1e96-11eb-9aff-16da7cad7b07","resourceVersion":"45488774","creationTimestamp":"2020-11-04T12:06:42Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"c5.xlarge","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"nodes","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-241.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":"","spotinst.io/node-lifecycle":"spot"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.111.0/24","providerID":"aws:///us-east-1f/i-091d1b1a0cf8a3486"},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"7807996Ki","pods":"58"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"7705596Ki","pods":"58"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:16Z","lastTransitionTime":"2020-11-04T12:06:42Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:16Z","lastTransitionTime":"2020-11-04T12:06:42Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:16Z","lastTransitionTime":"2020-11-04T12:06:42Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:53:16Z","lastTransitionTime":"2020-11-04T12:07:22Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.45.241"},{"type":"InternalDNS","address":"ip-10-**-**-241.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-241.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec24bda60e4af050185e3f297f0861eb","systemUUID":"EC24BDA6-0E4A-F050-185E-3F297F0861EB","bootID":"ce4d0187-9021-48d6-b39f-ea9771db9035","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.dev.example.com/cybervision/pregsense-user-management-service@sha256:e487a407cfaa96fa00f4d40a71e2792d26db1ad72843d9302588d18b0fb23326","registry.dev.example.com/cybervision/pregsense-user-management-service:2.0.0-37765"],"sizeBytes":1268651197},{"names":["registry.dev.example.com/ce/utilities/kafka-manager@sha256:c04a2fcec27aca7a0e8c70105ebd4c5f960ca81d6d867d4d584bcab9b7173fbc","registry.dev.example.com/ce/utilities/kafka-manager:2.0.0.2"],"sizeBytes":547394080},{"names":["registry.dev.example.com/ce/middleware/cassandra@sha256:29ff685aae75edaae14d052e723a66fabbd8e6e9ede068bb4a901776ada3b493","registry.dev.example.com/ce/middleware/cassandra:0.6.40"],"sizeBytes":384610888},{"names":["registry.dev.example.com/cybervision/pregsense-mobile-device-service@sha256:057469801785596922de4ad1866a4d2f7e26271d925889f1533fe8cf06b0db51","registry.dev.example.com/cybervision/pregsense-mobile-device-service:2.0.0-37577"],"sizeBytes":367804418},{"names":["registry.dev.example.com/cybervision/usability-data-extractor-service@sha256:e22df292989e525a9339b690d6d6a8e14134bb5ea6a06235691c019e11135f51","registry.dev.example.com/cybervision/usability-data-extractor-service:2.0.0-38072"],"sizeBytes":310003340},{"names":["registry.dev.example.com/cybervision/pregsense-medex-service@sha256:a6a38353b3b4ee41627606873b884cabf0a37bd0d6afb9b5f823b0e64d988d5b","registry.dev.example.com/cybervision/pregsense-medex-service:2.0.0-37573"],"sizeBytes":302750971},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["zookeeper@sha256:b7a76ec06f68fd9c801b72dfd283701bc7d8a8b0609277a0d570e8e6768e4ad9","zookeeper:3.5.5"],"sizeBytes":225210868},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["registry.dev.example.com/ce/utilities/cqlsh@sha256:36818e33bb74a5570efb254a468136d84f8520719e966927d0797ea512d9b073","registry.dev.example.com/ce/utilities/cqlsh:latest"],"sizeBytes":74739843},{"names":["redis@sha256:f8c22abc77f3f9cc1c2516062e4a2a71375859d7922da3faf9e4160e6ba4c3c2","redis:5.0.3-alpine"],"sizeBytes":50831292},{"names":["example/kube-tasks@sha256:713a62fab6a3075227d6a53681e548bc7cdaa39143bb135c939cc9573fd66965","example/kube-tasks:latest"],"sizeBytes":47161417},{"names":["registry.dev.example.com/ce/utilities/mysql-client@sha256:a748d356fc0b118b7a2cec5839fc73d473cfc9f8294c6ac8e07064c310e22ec9","registry.dev.example.com/ce/utilities/mysql-client:latest"],"sizeBytes":38388577},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["registry.dev.example.com/ce/utilities/kt@sha256:a52550f941a9449ea205201f2c42d0352da5f70bf281d1d638944352781851b2","registry.dev.example.com/ce/utilities/kt:latest"],"sizeBytes":22871156},{"names":["registry.dev.example.com/ce/observability/kafka-minion@sha256:34e6a52c9cef9b5d01f704c07913bd13b37a3465ae50d3adf56dd9a0ffd1aea5","registry.dev.example.com/ce/observability/kafka-minion:1.0.1"],"sizeBytes":19445889},{"names":["busybox@sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d","busybox:latest"],"sizeBytes":1231733},{"names":["busybox@sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0","busybox:1.30.0"],"sizeBytes":1199417},{"names":["busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47","busybox:1.28.4"],"sizeBytes":1146369},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}],"volumesInUse":["kubernetes.io/aws-ebs/aws://us-east-1f/vol-05a535855fa31264f","kubernetes.io/aws-ebs/aws://us-east-1f/vol-05f9ce80197f24f96","kubernetes.io/aws-ebs/aws://us-east-1f/vol-0b17552f50ccb204e"],"volumesAttached":[{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-0b17552f50ccb204e","devicePath":"/dev/xvdca"},{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-05a535855fa31264f","devicePath":"/dev/xvdch"},{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-05f9ce80197f24f96","devicePath":"/dev/xvdce"}]}},{"metadata":{"name":"ip-10-**-**-35.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-35.ec2.internal","uid":"1066fa28-1dfc-11eb-9aff-16da7cad7b07","resourceVersion":"45488746","creationTimestamp":"2020-11-03T17:43:20Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"r5.large","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"nodes","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-35.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":""},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.79.0/24","providerID":"aws:///us-east-1f/i-0e4fb331463f91e5b"},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"2","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16123244Ki","pods":"29"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"2","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16020844Ki","pods":"29"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:06Z","lastTransitionTime":"2020-11-03T17:43:20Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:06Z","lastTransitionTime":"2020-11-03T17:43:20Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:06Z","lastTransitionTime":"2020-11-03T17:43:20Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:53:06Z","lastTransitionTime":"2020-11-03T17:43:41Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.51.35"},{"type":"InternalDNS","address":"ip-10-**-**-35.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-35.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec21be1b38ff213ea283c19838e97f88","systemUUID":"EC21BE1B-38FF-213E-A283-C19838E97F88","bootID":"d7cb4873-93f4-4022-b59b-70fb30031938","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.dev.example.com/ce/utilities/kafka-shell@sha256:8a8a80dfaae72664413657ee2ea2bd4543f08163035bdefcb163456af9a8e316","registry.dev.example.com/ce/utilities/kafka-shell:latest"],"sizeBytes":697357066},{"names":["quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:617076c3e3d4d0638a4927702530d8456bda64c67194f6daed272a59e93b992f","quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0"],"sizeBytes":568075227},{"names":["registry.dev.example.com/ce/utilities/kafka-manager@sha256:c04a2fcec27aca7a0e8c70105ebd4c5f960ca81d6d867d4d584bcab9b7173fbc","registry.dev.example.com/ce/utilities/kafka-manager:2.0.0.2"],"sizeBytes":547394080},{"names":["registry.dev.example.com/ce/middleware/cassandra@sha256:29ff685aae75edaae14d052e723a66fabbd8e6e9ede068bb4a901776ada3b493","registry.dev.example.com/ce/middleware/cassandra:0.6.40"],"sizeBytes":384610888},{"names":["registry.dev.example.com/ce/middleware/mariadb@sha256:b4c12c27093137dbf047773d4ddb99676bedd32c8606bfa1e3c50876d5ce659f","registry.dev.example.com/ce/middleware/mariadb:0.6.56"],"sizeBytes":373763438},{"names":["registry.dev.example.com/cybervision/pregsense-mobile-device-service@sha256:057469801785596922de4ad1866a4d2f7e26271d925889f1533fe8cf06b0db51","registry.dev.example.com/cybervision/pregsense-mobile-device-service:2.0.0-37577"],"sizeBytes":367804418},{"names":["registry.dev.example.com/cybervision/pregsense-scheduler-service@sha256:7ad590d9311407a5c36583316cd79b66906db26085e56b5c37d9d3125d522fc2","registry.dev.example.com/cybervision/pregsense-scheduler-service:1.1.0-37581"],"sizeBytes":327939018},{"names":["registry.dev.example.com/cybervision/pregsense-medex-service@sha256:a6a38353b3b4ee41627606873b884cabf0a37bd0d6afb9b5f823b0e64d988d5b","registry.dev.example.com/cybervision/pregsense-medex-service:2.0.0-37573"],"sizeBytes":302750971},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["zookeeper@sha256:b7a76ec06f68fd9c801b72dfd283701bc7d8a8b0609277a0d570e8e6768e4ad9","zookeeper:3.5.5"],"sizeBytes":225210868},{"names":["registry.dev.example.com/ce/utilities/kubectl-in-pod@sha256:8b9d9dcedcc1eb0a3fa81971787469527333f0ee9aaf1eb296affba1da64a0e5","registry.dev.example.com/ce/utilities/kubectl-in-pod:0.0.4"],"sizeBytes":188973770},{"names":["registry.dev.example.com/cybervision/invu-web/support@sha256:de25604e5609c81e2d180b86126f00441671ab242555bda54a0b4e56d6a9b374","registry.dev.example.com/cybervision/invu-web/support:2.0.0-38041"],"sizeBytes":107592777},{"names":["gcr.io/kubernetes-helm/tiller@sha256:f8002b91997fdc2c15a9c2aa994bea117b5b1683933f3144369862f0883c3c42","gcr.io/kubernetes-helm/tiller:v2.14.1"],"sizeBytes":94194761},{"names":["emqx/emqx@sha256:3f31ee2793b4446e98183659b4493992355e2b0d8603df946c641b47f29ced78","emqx/emqx:v3.2.7"],"sizeBytes":89389623},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["registry.dev.example.com/ce/utilities/cqlsh@sha256:36818e33bb74a5570efb254a468136d84f8520719e966927d0797ea512d9b073","registry.dev.example.com/ce/utilities/cqlsh:latest"],"sizeBytes":74739843},{"names":["example/kube-tasks@sha256:713a62fab6a3075227d6a53681e548bc7cdaa39143bb135c939cc9573fd66965","example/kube-tasks:latest"],"sizeBytes":47161417},{"names":["k8s.gcr.io/coredns@sha256:2c8d61c46f484d881db43b34d13ca47a269336e576c81cf007ca740fa9ec0800","k8s.gcr.io/coredns:1.6.7"],"sizeBytes":43794147},{"names":["k8s.gcr.io/cluster-proportional-autoscaler-amd64@sha256:be8875e5584750b7a490244ee56a121a714aa3d124164a5090cd8b3570c5650f","k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1"],"sizeBytes":40684734},{"names":["k8s.gcr.io/metrics-server-amd64@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b","k8s.gcr.io/metrics-server-amd64:v0.3.6"],"sizeBytes":39944451},{"names":["registry.dev.example.com/ce/utilities/mysql-client@sha256:a748d356fc0b118b7a2cec5839fc73d473cfc9f8294c6ac8e07064c310e22ec9","registry.dev.example.com/ce/utilities/mysql-client:latest"],"sizeBytes":38388577},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["registry.dev.example.com/ce/middleware/connect-operator@sha256:7e5df82e5a2fc470d4f647d42b6849477ac712799868c87db99bdb88c778746f","registry.dev.example.com/ce/middleware/connect-operator:0.2.6"],"sizeBytes":32127859},{"names":["registry.dev.example.com/ce/observability/kafka-minion@sha256:34e6a52c9cef9b5d01f704c07913bd13b37a3465ae50d3adf56dd9a0ffd1aea5","registry.dev.example.com/ce/observability/kafka-minion:1.0.1"],"sizeBytes":19445889},{"names":["prom/mysqld-exporter@sha256:9f4fb61cca309cb4a8c1b9ed9fb4aa75af0f7a21f36d3954667db37c062a0172","prom/mysqld-exporter:v0.11.0"],"sizeBytes":17003848},{"names":["example/emq_exporter@sha256:1526c0476c1e5d740dfc4df36f3295ea27f87b41becf274c760fbd0a51011233","example/emq_exporter:v0.4.1"],"sizeBytes":11467840},{"names":["example/default-backend@sha256:14f15b1c7b037f32cc816e37259ffa386f2b0fdab5e48058b61323a389450f11","example/default-backend:latest"],"sizeBytes":5447342},{"names":["busybox@sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d","busybox:latest"],"sizeBytes":1231733},{"names":["busybox@sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0","busybox:1.30.0"],"sizeBytes":1199417},{"names":["busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47","busybox:1.28.4"],"sizeBytes":1146369},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}],"volumesInUse":["kubernetes.io/aws-ebs/aws://us-east-1f/vol-069646aaf149403ec","kubernetes.io/aws-ebs/aws://us-east-1f/vol-09a7d97ad0391bab1","kubernetes.io/aws-ebs/aws://us-east-1f/vol-0ebef74c040715d7b"],"volumesAttached":[{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-069646aaf149403ec","devicePath":"/dev/xvdch"},{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-0ebef74c040715d7b","devicePath":"/dev/xvdce"},{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-09a7d97ad0391bab1","devicePath":"/dev/xvdcg"}]}},{"metadata":{"name":"ip-10-**-**-156.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-156.ec2.internal","uid":"fafce029-0e0b-11eb-9aff-16da7cad7b07","resourceVersion":"45488711","creationTimestamp":"2020-10-14T10:56:58Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"m5a.large","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"master-us-east-1f","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-156.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"master","node-role.kubernetes.io/master":""},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.0.0/24","providerID":"aws:///us-east-1f/i-015c846b225ceb0dc","taints":[{"key":"node-role.kubernetes.io/master","effect":"NoSchedule"}]},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"2","ephemeral-storage":"64989720Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"7910764Ki","pods":"29"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"2","ephemeral-storage":"59894525853","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"7808364Ki","pods":"29"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:52:53Z","lastTransitionTime":"2020-10-14T10:56:58Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:52:53Z","lastTransitionTime":"2020-10-14T10:56:58Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:52:53Z","lastTransitionTime":"2020-10-14T10:56:58Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:52:53Z","lastTransitionTime":"2020-10-14T10:58:38Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.54.156"},{"type":"InternalDNS","address":"ip-10-**-**-156.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-156.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec2efbdeec6baae7f801227a04215e39","systemUUID":"EC2EFBDE-EC6B-AAE7-F801-227A04215E39","bootID":"41f6da35-f429-4543-bd6d-17c44e7c4cfc","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["spotinst/kubernetes-cluster-controller@sha256:f82c58b3e5374331ace6bf988987e2f0b390abedad313a31e5c8aaede3c580c2","spotinst/kubernetes-cluster-controller:1.0.68"],"sizeBytes":806421680},{"names":["kopeio/etcd-manager@sha256:08d90e866611142896c18c325d6b058d0aa0993f5b255807c17610a75cbe7392","kopeio/etcd-manager:3.0.20200531"],"sizeBytes":753910355},{"names":["spotinst/kubernetes-cluster-controller@sha256:28fb487f458ed0e2037cc7fe2e0d6a5df6ae095b5f4ad479cac07a0620d05ece","spotinst/kubernetes-cluster-controller:1.0.67"],"sizeBytes":740622913},{"names":["protokube:1.18.1"],"sizeBytes":285224997},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["k8s.gcr.io/kube-apiserver@sha256:223577e889dfb1a6e85faa6e3993308ead0b48b2a9c9d22c8fcd84b03d8a3af0","k8s.gcr.io/kube-apiserver:v1.14.2"],"sizeBytes":209916798},{"names":["k8s.gcr.io/kube-controller-manager@sha256:51a382e90acd9d11d5571850312ad4b268db8b28b2868516dfda19a6933a095c","k8s.gcr.io/kube-controller-manager:v1.14.2"],"sizeBytes":157933630},{"names":["kope/dns-controller@sha256:1df57184ec694b5a5839d2f59b54cc2a2857b34d94b3011063b3bc7caf8afb6d","kope/dns-controller:1.18.1"],"sizeBytes":89717254},{"names":["kope/dns-controller@sha256:ad0f2562cef00e77bb5c0ff255b682e25b14d24723e1e37057d349e39c7ff28d","kope/dns-controller:1.18.2"],"sizeBytes":89717254},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["k8s.gcr.io/kube-scheduler@sha256:052e0322b8a2b22819ab0385089f202555c4099493d1bd33205a34753494d2c2","k8s.gcr.io/kube-scheduler:v1.14.2"],"sizeBytes":81579742},{"names":["fluent/fluent-bit@sha256:5ff3eb13d23c38aac5aefcd0db5ea3562917d5c690b4b88257751667ac413ebd","fluent/fluent-bit:1.1.3"],"sizeBytes":53391352},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-iam-authenticator@sha256:1477d16f77b237d26d34f1cde1885931fbf98669c478a72e6849c1fcdc9365fe","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-iam-authenticator:v0.4.0"],"sizeBytes":23762101},{"names":["prom/node-exporter@sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee","prom/node-exporter:v0.18.1"],"sizeBytes":22933477},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}]}},{"metadata":{"name":"ip-10-**-**-41.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-41.ec2.internal","uid":"b4f79ec1-1e9b-11eb-9aff-16da7cad7b07","resourceVersion":"45488759","creationTimestamp":"2020-11-04T12:46:06Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"t3.xlarge","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"nodes","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-41.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":"","spotinst.io/node-lifecycle":"spot"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.113.0/24","providerID":"aws:///us-east-1f/i-07759285ebe2a969d"},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16235512Ki","pods":"58"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16133112Ki","pods":"58"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:09Z","lastTransitionTime":"2020-11-04T12:46:06Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:09Z","lastTransitionTime":"2020-11-04T12:46:06Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:09Z","lastTransitionTime":"2020-11-04T12:46:06Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:53:09Z","lastTransitionTime":"2020-11-04T12:46:27Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.55.41"},{"type":"InternalDNS","address":"ip-10-**-**-41.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-41.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec21154ae839a2204770777d165a898f","systemUUID":"EC21154A-E839-A220-4770-777D165A898F","bootID":"f2e8e581-1789-4138-bd5c-7c60dfa26880","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.dev.example.com/cybervision/invu-kafka-connect-sinks/kafka-connect@sha256:6a584826ecf0bbdf654fbe27201e2ed07d5586bd6f0c414c13be40ac3f6cc62f","registry.dev.example.com/cybervision/invu-kafka-connect-sinks/kafka-connect:2.0.0-37698"],"sizeBytes":643296613},{"names":["registry.dev.example.com/ce/middleware/cassandra@sha256:29ff685aae75edaae14d052e723a66fabbd8e6e9ede068bb4a901776ada3b493","registry.dev.example.com/ce/middleware/cassandra:0.6.40"],"sizeBytes":384610888},{"names":["registry.dev.example.com/cybervision/pregsense-device-service@sha256:ccdcd8bec0cc02d64da785a6ae863b31e335e28f089a92f071affaee5ec1a2cb","registry.dev.example.com/cybervision/pregsense-device-service:2.0.0-37554"],"sizeBytes":321951523},{"names":["registry.dev.example.com/ce/middleware/kafka@sha256:5a13ea153f26ac29e73fd3a0ed67c701f33ea761a0b10a03870fdda51e3975cc","registry.dev.example.com/ce/middleware/kafka:0.6.33"],"sizeBytes":269199283},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["registry.dev.example.com/ce/utilities/mysql-client@sha256:a748d356fc0b118b7a2cec5839fc73d473cfc9f8294c6ac8e07064c310e22ec9","registry.dev.example.com/ce/utilities/mysql-client:latest"],"sizeBytes":38388577},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["busybox@sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d","busybox:latest"],"sizeBytes":1231733},{"names":["busybox@sha256:7964ad52e396a6e045c39b5a44438424ac52e12e4d5a25d94895f2058cb863a0","busybox:1.30.0"],"sizeBytes":1199417},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}],"volumesInUse":["kubernetes.io/aws-ebs/aws://us-east-1f/vol-093bc9911e6b69ad5","kubernetes.io/aws-ebs/aws://us-east-1f/vol-0aa3e57d85128ef60"],"volumesAttached":[{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-093bc9911e6b69ad5","devicePath":"/dev/xvdbp"},{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-0aa3e57d85128ef60","devicePath":"/dev/xvdcc"}]}},{"metadata":{"name":"ip-10-**-**-27.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-27.ec2.internal","uid":"21d0f8a8-1e9b-11eb-9aff-16da7cad7b07","resourceVersion":"45488738","creationTimestamp":"2020-11-04T12:42:00Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"t3a.xlarge","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"nodes","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-27.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":"","spotinst.io/node-lifecycle":"spot"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.112.0/24","providerID":"aws:///us-east-1f/i-036e06b204f56f316"},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16280568Ki","pods":"58"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"4","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16178168Ki","pods":"58"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:03Z","lastTransitionTime":"2020-11-04T12:42:00Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:03Z","lastTransitionTime":"2020-11-04T12:42:00Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:03Z","lastTransitionTime":"2020-11-04T12:42:00Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:53:03Z","lastTransitionTime":"2020-11-04T12:42:50Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.56.27"},{"type":"InternalDNS","address":"ip-10-**-**-27.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-27.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec2fa63e40c54f665cbfe09ae4ec16d9","systemUUID":"EC2FA63E-40C5-4F66-5CBF-E09AE4EC16D9","bootID":"f2e9e82a-9b29-4591-a662-9fe3f838585e","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["confluentinc/cp-schema-registry@sha256:0d7a84d486477a0c203ae21d5d52c40d26e810f5b5cfa27a56274692b70949f5","confluentinc/cp-schema-registry:5.2.2"],"sizeBytes":746078624},{"names":["registry.dev.example.com/cybervision/invu-kafka-connect-sinks/kafka-connect@sha256:6a584826ecf0bbdf654fbe27201e2ed07d5586bd6f0c414c13be40ac3f6cc62f","registry.dev.example.com/cybervision/invu-kafka-connect-sinks/kafka-connect:2.0.0-37698"],"sizeBytes":643296613},{"names":["confluentinc/cp-ksql-server@sha256:a0b75d2f9926ac31139b2bebff3ac95a24e9f08465176aef16ed3c481b39a297","confluentinc/cp-ksql-server:5.2.2"],"sizeBytes":524657779},{"names":["registry.dev.example.com/ce/middleware/mariadb@sha256:b4c12c27093137dbf047773d4ddb99676bedd32c8606bfa1e3c50876d5ce659f","registry.dev.example.com/ce/middleware/mariadb:0.6.56"],"sizeBytes":373763438},{"names":["registry.dev.example.com/cybervision/invu-infrastructure-service@sha256:00f75459b2ee50deb47884131608f73108158d4faa085d1a5cc237aacd3f2112","registry.dev.example.com/cybervision/invu-infrastructure-service:2.0.0-37566"],"sizeBytes":312358195},{"names":["example/jmx-prometheus-httpserver@sha256:725e8a9bfec05f432b931edc11b1ee8bac3895e408bb91af58266dc88ee4e63b","example/jmx-prometheus-httpserver:0.11.0"],"sizeBytes":273954021},{"names":["registry.dev.example.com/ce/middleware/kafka@sha256:5a13ea153f26ac29e73fd3a0ed67c701f33ea761a0b10a03870fdda51e3975cc","registry.dev.example.com/ce/middleware/kafka:0.6.33"],"sizeBytes":269199283},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["emqx/emqx@sha256:3f31ee2793b4446e98183659b4493992355e2b0d8603df946c641b47f29ced78","emqx/emqx:v3.2.7"],"sizeBytes":89389623},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["registry.dev.example.com/ce/utilities/mysql-client@sha256:a748d356fc0b118b7a2cec5839fc73d473cfc9f8294c6ac8e07064c310e22ec9","registry.dev.example.com/ce/utilities/mysql-client:latest"],"sizeBytes":38388577},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["registry.dev.example.com/ce/utilities/kt@sha256:a52550f941a9449ea205201f2c42d0352da5f70bf281d1d638944352781851b2","registry.dev.example.com/ce/utilities/kt:latest"],"sizeBytes":22871156},{"names":["prom/mysqld-exporter@sha256:9f4fb61cca309cb4a8c1b9ed9fb4aa75af0f7a21f36d3954667db37c062a0172","prom/mysqld-exporter:v0.11.0"],"sizeBytes":17003848},{"names":["example/emq_exporter@sha256:1526c0476c1e5d740dfc4df36f3295ea27f87b41becf274c760fbd0a51011233","example/emq_exporter:v0.4.1"],"sizeBytes":11467840},{"names":["busybox@sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d","busybox:latest"],"sizeBytes":1231733},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}],"volumesInUse":["kubernetes.io/aws-ebs/aws://us-east-1f/vol-0153f8347ff8a2cd1","kubernetes.io/aws-ebs/aws://us-east-1f/vol-0cb2890c3adac40d4"],"volumesAttached":[{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-0153f8347ff8a2cd1","devicePath":"/dev/xvdbj"},{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-0cb2890c3adac40d4","devicePath":"/dev/xvdcd"}]}},{"metadata":{"name":"ip-10-**-**-245.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-245.ec2.internal","uid":"b7917ff0-1e95-11eb-9aff-16da7cad7b07","resourceVersion":"45488699","creationTimestamp":"2020-11-04T12:03:14Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"t3a.medium","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"nodes","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-245.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":"","spotinst.io/node-lifecycle":"spot"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.107.0/24","providerID":"aws:///us-east-1f/i-0deead86f7d952607"},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"2","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"3990892Ki","pods":"17"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"2","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"3888492Ki","pods":"17"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:52:49Z","lastTransitionTime":"2020-11-04T12:03:14Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:52:49Z","lastTransitionTime":"2020-11-04T12:03:14Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:52:49Z","lastTransitionTime":"2020-11-04T12:03:14Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:52:49Z","lastTransitionTime":"2020-11-04T12:03:34Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.57.245"},{"type":"InternalDNS","address":"ip-10-**-**-245.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-245.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec2d967bf15437edf65a244bceea6ecc","systemUUID":"EC2D967B-F154-37ED-F65A-244BCEEA6ECC","bootID":"924eea63-f8c3-4e69-8932-c2622c6c300d","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.dev.example.com/cybervision/pregsense-gateway-service@sha256:ba2b7c16fd295c73ab75edee8808ffd8758c01ad0bd74d394aa7b3395f61d4ad","registry.dev.example.com/cybervision/pregsense-gateway-service:2.0.0-38164"],"sizeBytes":339009338},{"names":["registry.dev.example.com/cybervision/pregsense-auth-service@sha256:96526bbbd74cffab866d7cdf38ccc23cc59228a95d6d6c10d28f4ee056f7f4d0","registry.dev.example.com/cybervision/pregsense-auth-service:2.0.0-37888"],"sizeBytes":333981635},{"names":["registry.dev.example.com/cybervision/pregsense-activity-service@sha256:52a50e96512249156c779d18a5aecc8082e2829221ecd3bc635401bf76ab9c61","registry.dev.example.com/cybervision/pregsense-activity-service:2.0.0-37549"],"sizeBytes":325759251},{"names":["registry.dev.example.com/cybervision/pregsense-geo-service@sha256:9e44ea04116b0dad47c7774c04666e6a74de49e252ebd2b3f3d562ce63bf9bbc","registry.dev.example.com/cybervision/pregsense-geo-service:2.0.0-37561"],"sizeBytes":298502628},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["zookeeper@sha256:b7a76ec06f68fd9c801b72dfd283701bc7d8a8b0609277a0d570e8e6768e4ad9","zookeeper:3.5.5"],"sizeBytes":225210868},{"names":["emqx/emqx@sha256:3f31ee2793b4446e98183659b4493992355e2b0d8603df946c641b47f29ced78","emqx/emqx:v3.2.7"],"sizeBytes":89389623},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["registry.dev.example.com/ce/utilities/mysql-client@sha256:a748d356fc0b118b7a2cec5839fc73d473cfc9f8294c6ac8e07064c310e22ec9","registry.dev.example.com/ce/utilities/mysql-client:latest"],"sizeBytes":38388577},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["example/emq_exporter@sha256:1526c0476c1e5d740dfc4df36f3295ea27f87b41becf274c760fbd0a51011233","example/emq_exporter:v0.4.1"],"sizeBytes":11467840},{"names":["busybox@sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d","busybox:latest"],"sizeBytes":1231733},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}],"volumesInUse":["kubernetes.io/aws-ebs/aws://us-east-1f/vol-032d1079efa034f3c"],"volumesAttached":[{"name":"kubernetes.io/aws-ebs/aws://us-east-1f/vol-032d1079efa034f3c","devicePath":"/dev/xvdcx"}]}},{"metadata":{"name":"ip-10-**-**-224.ec2.internal","selfLink":"/api/v1/nodes/ip-10-**-**-224.ec2.internal","uid":"555d2c07-1ea0-11eb-9aff-16da7cad7b07","resourceVersion":"45488855","creationTimestamp":"2020-11-04T13:19:14Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"t3a.2xlarge","beta.kubernetes.io/os":"linux","failure-domain.beta.kubernetes.io/region":"us-east-1","failure-domain.beta.kubernetes.io/zone":"us-east-1f","kops.k8s.io/instancegroup":"nodes","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"ip-10-**-**-224.ec2.internal","kubernetes.io/os":"linux","kubernetes.io/role":"node","node-role.kubernetes.io/node":"","spotinst.io/node-lifecycle":"spot"},"annotations":{"node.alpha.kubernetes.io/ttl":"0","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"100.96.117.0/24","providerID":"aws:///us-east-1f/i-0ffadee0802927c01"},"status":{"capacity":{"attachable-volumes-aws-ebs":"25","cpu":"8","ephemeral-storage":"130045936Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"32665868Ki","pods":"58"},"allocatable":{"attachable-volumes-aws-ebs":"25","cpu":"8","ephemeral-storage":"119850334420","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"32563468Ki","pods":"58"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:45Z","lastTransitionTime":"2020-11-04T13:19:14Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:45Z","lastTransitionTime":"2020-11-04T13:19:14Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2020-11-04T13:53:45Z","lastTransitionTime":"2020-11-04T13:19:14Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2020-11-04T13:53:45Z","lastTransitionTime":"2020-11-04T13:20:04Z","reason":"KubeletReady","message":"kubelet is posting ready status. AppArmor enabled"}],"addresses":[{"type":"InternalIP","address":"10.40.59.224"},{"type":"InternalDNS","address":"ip-10-**-**-224.ec2.internal"},{"type":"Hostname","address":"ip-10-**-**-224.ec2.internal"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"ec2d3362029642b9b8a2cb822c33b5db","systemUUID":"EC2D3362-0296-42B9-B8A2-CB822C33B5DB","bootID":"25c585bc-acbd-4a0e-bbe1-a99b8f545565","kernelVersion":"4.15.0-1039-aws","osImage":"Ubuntu 18.04.2 LTS","containerRuntimeVersion":"docker://18.6.3","kubeletVersion":"v1.14.2","kubeProxyVersion":"v1.14.2","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.dev.example.com/cybervision/pregsense-user-management-service@sha256:e487a407cfaa96fa00f4d40a71e2792d26db1ad72843d9302588d18b0fb23326","registry.dev.example.com/cybervision/pregsense-user-management-service:2.0.0-37765"],"sizeBytes":1268651197},{"names":["registry.dev.example.com/cybervision/usability-data-extractor-service@sha256:e22df292989e525a9339b690d6d6a8e14134bb5ea6a06235691c019e11135f51","registry.dev.example.com/cybervision/usability-data-extractor-service:2.0.0-38072"],"sizeBytes":310003340},{"names":["602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni@sha256:a6d23b9fb3d4ba549321e32a28c42d8e79da203897072e93874472ab9e80b768","602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.5.5"],"sizeBytes":263850871},{"names":["k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc","k8s.gcr.io/kube-proxy:v1.14.2"],"sizeBytes":82106236},{"names":["registry.dev.example.com/ce/utilities/mysql-client@sha256:a748d356fc0b118b7a2cec5839fc73d473cfc9f8294c6ac8e07064c310e22ec9","registry.dev.example.com/ce/utilities/mysql-client:latest"],"sizeBytes":38388577},{"names":["jtblin/kube2iam@sha256:92363f1c52fe5f433487b35c3f9c242e4733717ef7ac88c7137fc3824cfad4d3","jtblin/kube2iam:0.10.8"],"sizeBytes":34303129},{"names":["k8s.gcr.io/pause-amd64@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108","k8s.gcr.io/pause-amd64:3.2"],"sizeBytes":682696}]}}]}
I1104 15:53:47.584459 12397 s3fs.go:327] Listing objects in S3 bucket "some-bucket" with prefix "kops/test.example.com/instancegroup/"
I1104 15:53:47.816103 12397 s3fs.go:355] Listed files in s3://some-bucket/kops/test.example.com/instancegroup: [s3://some-bucket/kops/test.example.com/instancegroup/another-group s3://some-bucket/kops/test.example.com/instancegroup/bastions s3://some-bucket/kops/test.example.com/instancegroup/master-us-east-1f s3://some-bucket/kops/test.example.com/instancegroup/nodes]
I1104 15:53:47.816179 12397 s3fs.go:290] Reading file "s3://some-bucket/kops/test.example.com/instancegroup/another-group"
I1104 15:53:48.075318 12397 s3fs.go:290] Reading file "s3://some-bucket/kops/test.example.com/instancegroup/bastions"
I1104 15:53:48.277040 12397 s3fs.go:290] Reading file "s3://some-bucket/kops/test.example.com/instancegroup/master-us-east-1f"
I1104 15:53:48.483285 12397 s3fs.go:290] Reading file "s3://some-bucket/kops/test.example.com/instancegroup/nodes"
I1104 15:53:48.690796 12397 aws_cloud.go:1340] Querying EC2 for all valid zones in region "us-east-1"
I1104 15:53:48.891261 12397 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I1104 15:53:49.659735 12397 resources.go:36] Listing all resources
I1104 15:53:49.659792 12397 resources.go:58] Listing all Elastigroup resources
I1104 15:53:49.660173 12397 spotinst.go:80] SPOTINST: Request "GET https://api.spotinst.io/aws/ec2/group?accountId=act-*******" details:
---[ REQUEST ]---------------------------------------
GET /aws/ec2/group?accountId=act-******* HTTP/1.1
Host: api.spotinst.io
User-Agent: kubernetes-kops/1.18.2 spotinst-sdk-go/1.49.0 (go1.13.9; linux; amd64)
Accept: application/json
Authorization: Bearer ****
Content-Type: application/json
Accept-Encoding: gzip
-----------------------------------------------------
I1104 15:53:51.251990 12397 spotinst.go:80] SPOTINST: Response "GET https://api.spotinst.io/aws/ec2/group?accountId=act-*******" details:
---[ RESPONSE ]----------------------------------------
HTTP/1.1 200 OK
Content-Length: 139539
Access-Control-Allow-Headers: Origin,Accept,Content-Type,Authorization,X-CSRF-Token
Access-Control-Allow-Methods: GET,POST,DELETE,PUT
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Wed, 04 Nov 2020 13:53:50 GMT
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Ratelimit-Limit: 300
X-Ratelimit-Remaining: 299
X-Request-Id: 50057f88-e411-4ead-b894-e9047f4e868e
X-Response-Time: 215ms
{
"request": {
"id": "50057f88-e411-4ead-b894-e9047f4e868e",
"url": "/aws/ec2/group",
"method": "GET",
"timestamp": "2020-11-04T13:53:50.553Z"
},
"response": {
"status": {
"code": 200,
"message": "OK"
},
"kind": "spotinst:aws:ec2:group",
"items": [
{
"id": "sig-********",
"name": "master-us-east-1c.masters.dev.example.com",
"description": "master-us-east-1c.masters.dev.example.com",
"capacity": {
"minimum": 1,
"maximum": 1,
"target": 1,
"unit": "instance"
},
"strategy": {
"risk": 0,
"onDemandCount": null,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {}
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5.large",
"m4.large"
]
},
"availabilityZones": [
{
"name": "us-east-1c",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": [
{
"name": "api-dev-example-com-vo1g1r",
"type": "CLASSIC"
}
]
},
"securityGroupIds": [
"sg-0c04097ebbe0cfa7f"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "masters.dev.example.com",
"arn": null
},
"keyPair": "kubernetes.dev.example.com-************************",
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 64,
"volumeType": "GP2"
}
}
],
"tags": [
{
"tagKey": "Name",
"tagValue": "master-us-east-1c.masters.dev.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "dev.example.com"
},
{
"tagKey": "kubernetes.io/cluster/dev.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "dev"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1c"
},
{
"tagKey": "k8s.io/role/master",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1c"
}
]
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "dev.example.com",
"autoScale": {
"isEnabled": false,
"isAutoConfig": true
}
}
},
"multai": null,
"createdAt": "2019-04-14T07:44:27.000Z",
"updatedAt": "2020-10-14T10:02:58.000Z"
},
{
"id": "sig-********",
"name": "master-us-east-1f.masters.dev.example.com",
"description": "master-us-east-1f.masters.dev.example.com",
"capacity": {
"minimum": 1,
"maximum": 1,
"target": 1,
"unit": "instance"
},
"strategy": {
"risk": 0,
"onDemandCount": null,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {}
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5.large",
"m4.large"
]
},
"availabilityZones": [
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": [
{
"name": "api-dev-example-com-vo1g1r",
"type": "CLASSIC"
}
]
},
"securityGroupIds": [
"sg-0c04097ebbe0cfa7f"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "masters.dev.example.com",
"arn": null
},
"keyPair": "kubernetes.dev.example.com-************************",
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 64,
"volumeType": "GP2"
}
}
],
"tags": [
{
"tagKey": "KubernetesCluster",
"tagValue": "dev.example.com"
},
{
"tagKey": "kubernetes.io/cluster/dev.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "dev"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1f"
},
{
"tagKey": "k8s.io/role/master",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1f"
},
{
"tagKey": "Name",
"tagValue": "master-us-east-1f.masters.dev.example.com"
}
]
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "dev.example.com",
"autoScale": {
"isEnabled": false,
"down": {
"evaluationPeriods": 3
},
"headroom": {
"cpuPerUnit": 0,
"memoryPerUnit": 0,
"gpuPerUnit": 0,
"numOfUnits": 0
},
"isAutoConfig": false
}
}
},
"multai": null,
"createdAt": "2019-04-14T07:44:38.000Z",
"updatedAt": "2020-10-14T10:07:30.000Z"
},
{
"id": "sig-********",
"name": "bastions.dev.example.com",
"description": "bastions.dev.example.com",
"capacity": {
"minimum": 1,
"maximum": 1,
"target": 1,
"unit": "instance"
},
"strategy": {
"risk": 100,
"onDemandCount": null,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"lifetimePeriod": "days",
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {},
"persistence": {},
"revertToSpot": {
"performAt": "always"
}
},
"compute": {
"instanceTypes": {
"ondemand": "t3a.nano",
"spot": [
"t3a.nano",
"t3.nano",
"t3a.micro",
"t3.micro",
"t2.micro",
"t3a.small",
"t3.small",
"t2.small"
]
},
"availabilityZones": [
{
"name": "us-east-1b",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1c",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": [
{
"name": "bastion-dev-example-com-mknckd",
"type": "CLASSIC"
}
]
},
"healthCheckType": null,
"securityGroupIds": [
"sg-0c03575c501eeb4f8"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "bastions.dev.example.com",
"arn": null
},
"keyPair": "kubernetes.dev.example.com-************************",
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 32,
"volumeType": "GP2"
}
}
],
"networkInterfaces": [
{
"deviceIndex": 0,
"associatePublicIpAddress": true,
"deleteOnTermination": true,
"associateIpv6Address": false
}
],
"tags": [
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "bastions"
},
{
"tagKey": "Name",
"tagValue": "bastions.dev.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "dev.example.com"
},
{
"tagKey": "kubernetes.io/cluster/dev.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "dev"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "bastions"
},
{
"tagKey": "k8s.io/role/bastion",
"tagValue": "1"
}
],
"tenancy": "default"
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {},
"multai": null,
"createdAt": "2019-04-14T07:44:38.000Z",
"updatedAt": "2020-10-11T10:59:26.000Z"
},
{
"id": "sig-********",
"name": "r-nodes.dev.example.com",
"description": "r-nodes.dev.example.com",
"capacity": {
"minimum": 5,
"maximum": 30,
"target": 21,
"unit": "instance"
},
"strategy": {
"risk": 0,
"onDemandCount": 40,
"availabilityVsCost": "equalAzDistribution",
"drainingTimeout": 120,
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {}
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5a.xlarge",
"m5a.2xlarge",
"m5.large",
"m5.xlarge",
"m5.2xlarge",
"m4.large",
"m4.xlarge",
"m4.2xlarge",
"t3.micro",
"t3a.micro",
"t3.small",
"t3a.small",
"t3.medium",
"t3a.medium",
"t3.large",
"t3a.large",
"t3.xlarge",
"t3a.xlarge",
"t3.2xlarge",
"t3a.2xlarge",
"r5a.large",
"r5a.xlarge",
"r5a.2xlarge",
"r5.large",
"r5.xlarge",
"r5.2xlarge",
"c5.large",
"c5.xlarge",
"c5.2xlarge"
]
},
"availabilityZones": [
{
"name": "us-east-1b",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1c",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": null
},
"healthCheckType": "K8S_NODE",
"securityGroupIds": [
"sg-0655cde58fc563d18"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "nodes.dev.example.com",
"arn": null
},
"keyPair": "kubernetes.dev.example.com-************************",
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 128,
"volumeType": "GP2"
}
}
],
"networkInterfaces": [
{
"description": "eth0",
"deviceIndex": 0,
"associatePublicIpAddress": false,
"deleteOnTermination": true,
"associateIpv6Address": false
}
],
"tags": [
{
"tagKey": "env",
"tagValue": "dev"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "r-nodes"
},
{
"tagKey": "k8s.io/role/node",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "r-nodes"
},
{
"tagKey": "Name",
"tagValue": "r-nodes.dev.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "dev.example.com"
},
{
"tagKey": "kubernetes.io/cluster/dev.example.com",
"tagValue": "owned"
}
]
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "dev.example.com",
"autoScale": {
"isEnabled": true,
"cooldown": 300,
"headroom": {
"cpuPerUnit": 0,
"memoryPerUnit": 0,
"gpuPerUnit": 0,
"numOfUnits": 0
},
"isAutoConfig": false
}
}
},
"multai": null,
"createdAt": "2020-01-07T09:51:47.000Z",
"updatedAt": "2020-11-04T11:37:12.000Z"
},
{
"id": "sig-********",
"name": "another-group.dev.example.com",
"description": "another-group.dev.example.com",
"capacity": {
"minimum": 1,
"maximum": 8,
"target": 3,
"unit": "instance"
},
"strategy": {
"risk": 0,
"onDemandCount": 60,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {}
},
"compute": {
"instanceTypes": {
"ondemand": "c5.2xlarge",
"spot": [
"c5.2xlarge"
]
},
"availabilityZones": [
{
"name": "us-east-1b",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1c",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": null
},
"securityGroupIds": [
"sg-0655cde58fc563d18"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "nodes.dev.example.com",
"arn": null
},
"keyPair": "kubernetes.dev.example.com-************************",
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 128,
"volumeType": "GP2"
}
}
],
"tags": [
{
"tagKey": "KubernetesCluster",
"tagValue": "dev.example.com"
},
{
"tagKey": "kubernetes.io/cluster/dev.example.com",
"tagValue": "owned"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/dedicated",
"tagValue": "another-group"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "another-group"
},
{
"tagKey": "Name",
"tagValue": "another-group.dev.example.com"
},
{
"tagKey": "env",
"tagValue": "dev"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/taint/dedicated",
"tagValue": "another-group:NoSchedule"
},
{
"tagKey": "k8s.io/role/node",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "another-group"
}
]
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "dev.example.com",
"autoScale": {
"isEnabled": true,
"cooldown": 300,
"headroom": {
"cpuPerUnit": 0,
"memoryPerUnit": 0,
"gpuPerUnit": 0,
"numOfUnits": 0
},
"labels": [
{
"key": "dedicated",
"value": "another-group"
}
],
"isAutoConfig": false
}
}
},
"multai": null,
"createdAt": "2019-04-14T07:43:51.000Z",
"updatedAt": "2020-11-04T09:42:12.000Z"
},
{
"id": "sig-********",
"name": "dev-tools.dev.example.com",
"description": "dev-tools.dev.example.com",
"capacity": {
"minimum": 3,
"maximum": 5,
"target": 3,
"unit": "instance"
},
"strategy": {
"risk": 100,
"onDemandCount": 3,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {}
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5a.xlarge",
"m5a.2xlarge",
"m5.large",
"m5.xlarge",
"m5.2xlarge",
"m4.large",
"m4.xlarge",
"m4.2xlarge",
"t3.micro",
"t3a.micro",
"t3.small",
"t3a.small",
"t3.medium",
"t3a.medium",
"t3.large",
"t3a.large",
"t3.xlarge",
"t3a.xlarge",
"t3.2xlarge",
"t3a.2xlarge",
"r5a.large",
"r5a.xlarge",
"r5a.2xlarge",
"r5.large",
"r5.xlarge",
"r5.2xlarge",
"c5.large",
"c5.xlarge",
"c5.2xlarge"
]
},
"availabilityZones": [
{
"name": "us-east-1b",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1c",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
},
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": null
},
"securityGroupIds": [
"sg-0655cde58fc563d18"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "nodes.dev.example.com",
"arn": null
},
"keyPair": "kubernetes.dev.example.com-************************",
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 128,
"volumeType": "GP2"
}
}
],
"tags": [
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/dedicated",
"tagValue": "dev-tools"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/taint/dedicated",
"tagValue": "dev-tools:NoSchedule"
},
{
"tagKey": "k8s.io/role/node",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "dev-tools"
},
{
"tagKey": "Name",
"tagValue": "dev-tools.dev.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "dev.example.com"
},
{
"tagKey": "env",
"tagValue": "dev"
},
{
"tagKey": "kubernetes.io/cluster/dev.example.com",
"tagValue": "owned"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "dev-tools"
}
]
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "dev.example.com",
"autoScale": {
"isEnabled": true,
"cooldown": 300,
"headroom": {
"cpuPerUnit": 0,
"memoryPerUnit": 0,
"gpuPerUnit": 0,
"numOfUnits": 0
},
"labels": [
{
"key": "dedicated",
"value": "dev-tools"
}
],
"isAutoConfig": false
}
}
},
"multai": null,
"createdAt": "2019-04-14T07:43:58.000Z",
"updatedAt": "2020-10-12T20:04:14.000Z"
},
{
"id": "sig-********",
"name": "bastions.test.example.com",
"description": "bastions.test.example.com",
"capacity": {
"minimum": 1,
"maximum": 1,
"target": 1,
"unit": "instance"
},
"strategy": {
"risk": 100,
"availabilityVsCost": "balanced",
"utilizeReservedInstances": true,
"fallbackToOd": true
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5a.xlarge",
"m5a.2xlarge",
"m5.large",
"m5.xlarge",
"m5.2xlarge",
"m4.large",
"m4.xlarge",
"m4.2xlarge",
"t3.small",
"t3a.small",
"t3.medium",
"t3a.medium",
"t3.large",
"t3a.large",
"t3.xlarge",
"t3a.xlarge",
"t3.2xlarge",
"t3a.2xlarge",
"r5a.large",
"r5a.xlarge",
"r5a.2xlarge",
"r5.large",
"r5.xlarge",
"r5.2xlarge",
"c5.large",
"c5.xlarge",
"c5.2xlarge",
"c5a.large",
"c5a.xlarge",
"c5a.2xlarge"
]
},
"availabilityZones": [
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancersConfig": {
"loadBalancers": [
{
"name": "bastion-test-example-com-ogksqp",
"type": "CLASSIC"
}
]
},
"healthCheckType": "K8S_NODE",
"securityGroupIds": [
"sg-0e55267107f6e0e84"
],
"monitoring": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "bastions.test.example.com",
"arn": null
},
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 32,
"volumeType": "GP2"
}
}
],
"networkInterfaces": [
{
"description": "eth0",
"deviceIndex": 0,
"associatePublicIpAddress": true,
"deleteOnTermination": true,
"associateIpv6Address": false
}
],
"tags": [
{
"tagKey": "k8s.io/role/bastion",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "bastions"
},
{
"tagKey": "Name",
"tagValue": "bastions.test.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "test.example.com"
},
{
"tagKey": "kubernetes.io/cluster/test.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "test"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "bastions"
}
]
}
},
"createdAt": "2019-11-19T16:49:13.000Z",
"updatedAt": "2020-10-14T11:37:54.000Z"
},
{
"id": "sig-********",
"name": "nodes.test.example.com",
"description": "nodes.test.example.com",
"capacity": {
"minimum": 0,
"maximum": 17,
"target": 7,
"unit": "instance"
},
"strategy": {
"risk": 100,
"onDemandCount": null,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"lifetimePeriod": "days",
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {},
"persistence": {},
"revertToSpot": {
"performAt": "always"
}
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5a.xlarge",
"m5a.2xlarge",
"m5.large",
"m5.xlarge",
"m5.2xlarge",
"m4.large",
"m4.xlarge",
"m4.2xlarge",
"t3.small",
"t3a.small",
"t3.medium",
"t3a.medium",
"t3.large",
"t3a.large",
"t3.xlarge",
"t3a.xlarge",
"t3.2xlarge",
"t3a.2xlarge",
"r5a.large",
"r5a.xlarge",
"r5a.2xlarge",
"r5.large",
"r5.xlarge",
"r5.2xlarge",
"c5.large",
"c5.xlarge",
"c5.2xlarge",
"c5a.large",
"c5a.xlarge",
"c5a.2xlarge"
]
},
"availabilityZones": [
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"preferredAvailabilityZones": null,
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": null
},
"healthCheckType": "K8S_NODE",
"healthCheckGracePeriod": 600,
"securityGroupIds": [
"sg-0300a03f04c61b109"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "nodes.test.example.com",
"arn": null
},
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 128,
"volumeType": "GP2"
}
}
],
"networkInterfaces": [
{
"description": "eth0",
"deviceIndex": 0,
"associatePublicIpAddress": false,
"deleteOnTermination": true,
"associateIpv6Address": false
}
],
"tags": [
{
"tagKey": "kubernetes.io/cluster/test.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "test"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "nodes"
},
{
"tagKey": "k8s.io/role/node",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "nodes"
},
{
"tagKey": "Name",
"tagValue": "nodes.test.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "test.example.com"
}
],
"tenancy": "default"
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "test.example.com",
"autoScale": {
"isEnabled": true,
"cooldown": 300,
"headroom": {
"cpuPerUnit": 0,
"memoryPerUnit": 0,
"gpuPerUnit": 0,
"numOfUnits": 0
},
"isAutoConfig": false
}
}
},
"multai": null,
"createdAt": "2019-11-19T16:48:56.000Z",
"updatedAt": "2020-11-04T13:17:13.000Z"
},
{
"id": "sig-********",
"name": "another-group.test.example.com",
"description": "another-group.test.example.com",
"capacity": {
"minimum": 0,
"maximum": 8,
"target": 1,
"unit": "instance"
},
"strategy": {
"risk": 100,
"availabilityVsCost": "balanced",
"utilizeReservedInstances": true,
"fallbackToOd": true
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5a.xlarge",
"m5a.2xlarge",
"m5.large",
"m5.xlarge",
"m5.2xlarge",
"m4.large",
"m4.xlarge",
"m4.2xlarge",
"t3.small",
"t3a.small",
"t3.medium",
"t3a.medium",
"t3.large",
"t3a.large",
"t3.xlarge",
"t3a.xlarge",
"t3.2xlarge",
"t3a.2xlarge",
"r5a.large",
"r5a.xlarge",
"r5a.2xlarge",
"r5.large",
"r5.xlarge",
"r5.2xlarge",
"c5.large",
"c5.xlarge",
"c5.2xlarge",
"c5a.large",
"c5a.xlarge",
"c5a.2xlarge"
]
},
"availabilityZones": [
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"securityGroupIds": [
"sg-0300a03f04c61b109"
],
"monitoring": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "nodes.test.example.com",
"arn": null
},
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 128,
"volumeType": "GP2"
}
}
],
"tags": [
{
"tagKey": "KubernetesCluster",
"tagValue": "test.example.com"
},
{
"tagKey": "kubernetes.io/cluster/test.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "test"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/dedicated",
"tagValue": "another-group"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "another-group"
},
{
"tagKey": "k8s.io/role/node",
"tagValue": "1"
},
{
"tagKey": "Name",
"tagValue": "another-group.test.example.com"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/taint/dedicated",
"tagValue": "another-group:NoSchedule"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "another-group"
}
]
}
},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "test.example.com",
"autoScale": {
"isEnabled": true,
"cooldown": 300,
"headroom": {
"cpuPerUnit": 0,
"memoryPerUnit": 0,
"gpuPerUnit": 0,
"numOfUnits": 0
},
"labels": [
{
"key": "dedicated",
"value": "another-group"
}
],
"isAutoConfig": false
}
}
},
"createdAt": "2020-08-18T11:14:10.000Z",
"updatedAt": "2020-11-04T13:17:14.000Z"
},
{
"id": "sig-********",
"name": "master-us-east-1b.masters.dev.example.com",
"description": "master-us-east-1b.masters.dev.example.com",
"capacity": {
"minimum": 1,
"maximum": 1,
"target": 1,
"unit": "instance"
},
"strategy": {
"risk": 0,
"onDemandCount": null,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {}
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5.large",
"m4.large"
]
},
"availabilityZones": [
{
"name": "us-east-1b",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": [
{
"name": "api-dev-example-com-vo1g1r",
"type": "CLASSIC"
}
]
},
"securityGroupIds": [
"sg-0c04097ebbe0cfa7f"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "masters.dev.example.com",
"arn": null
},
"keyPair": "kubernetes.dev.example.com-************************",
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 64,
"volumeType": "GP2"
}
}
],
"tags": [
{
"tagKey": "kubernetes.io/cluster/dev.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "dev"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1b"
},
{
"tagKey": "k8s.io/role/master",
"tagValue": "1"
},
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1b"
},
{
"tagKey": "Name",
"tagValue": "master-us-east-1b.masters.dev.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "dev.example.com"
}
]
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "dev.example.com",
"autoScale": {
"isEnabled": false,
"down": {
"evaluationPeriods": 3
},
"headroom": {
"cpuPerUnit": 0,
"memoryPerUnit": 0,
"gpuPerUnit": 0,
"numOfUnits": 0
},
"isAutoConfig": false
}
}
},
"multai": null,
"createdAt": "2019-04-14T07:44:38.000Z",
"updatedAt": "2020-10-14T10:08:27.000Z"
},
{
"id": "sig-********",
"name": "master-us-east-1f.masters.test.example.com",
"description": "master-us-east-1f.masters.test.example.com",
"capacity": {
"minimum": 1,
"maximum": 1,
"target": 1,
"unit": "instance"
},
"strategy": {
"risk": 0,
"onDemandCount": null,
"availabilityVsCost": "balanced",
"drainingTimeout": 120,
"utilizeReservedInstances": true,
"fallbackToOd": true,
"scalingStrategy": {}
},
"compute": {
"instanceTypes": {
"ondemand": "m5a.large",
"spot": [
"m5a.large",
"m5a.xlarge",
"m5a.2xlarge",
"m5.large",
"m5.xlarge",
"m5.2xlarge",
"m4.large",
"m4.xlarge",
"m4.2xlarge",
"t3.small",
"t3a.small",
"t3.medium",
"t3a.medium",
"t3.large",
"t3a.large",
"t3.xlarge",
"t3a.xlarge",
"t3.2xlarge",
"t3a.2xlarge",
"r5a.large",
"r5a.xlarge",
"r5a.2xlarge",
"r5.large",
"r5.xlarge",
"r5.2xlarge",
"c5.large",
"c5.xlarge",
"c5.2xlarge",
"c5a.large",
"c5a.xlarge",
"c5a.2xlarge"
]
},
"availabilityZones": [
{
"name": "us-east-1f",
"subnetIds": [
"subnet-**************"
],
"subnetId": "subnet-**************"
}
],
"product": "Linux/UNIX",
"launchSpecification": {
"loadBalancerNames": null,
"loadBalancersConfig": {
"loadBalancers": [
{
"name": "api-test-example-com-ps2tfr",
"type": "CLASSIC"
}
]
},
"healthCheckType": "K8S_NODE",
"securityGroupIds": [
"sg-0ea9c09b1cf6d2246"
],
"monitoring": false,
"ebsOptimized": false,
"imageId": "ami-024a64a6685d05041",
"iamRole": {
"name": "masters.test.example.com",
"arn": null
},
"shutdownScript": null,
"blockDeviceMappings": [
{
"deviceName": "/dev/sda1",
"ebs": {
"deleteOnTermination": true,
"volumeSize": 64,
"volumeType": "GP2"
}
}
],
"networkInterfaces": [
{
"description": "eth0",
"deviceIndex": 0,
"associatePublicIpAddress": false,
"deleteOnTermination": true,
"associateIpv6Address": false
}
],
"tags": [
{
"tagKey": "kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1f"
},
{
"tagKey": "Name",
"tagValue": "master-us-east-1f.masters.test.example.com"
},
{
"tagKey": "KubernetesCluster",
"tagValue": "test.example.com"
},
{
"tagKey": "kubernetes.io/cluster/test.example.com",
"tagValue": "owned"
},
{
"tagKey": "env",
"tagValue": "test"
},
{
"tagKey": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
"tagValue": "master-us-east-1f"
},
{
"tagKey": "k8s.io/role/master",
"tagValue": "1"
}
]
}
},
"scaling": {},
"scheduling": {},
"thirdPartiesIntegration": {
"kubernetes": {
"integrationMode": "pod",
"clusterIdentifier": "test.example.com",
"autoScale": {
"isEnabled": false,
"cooldown": 300,
"isAutoConfig": true
}
}
},
"multai": null,
"createdAt": "2019-11-19T16:49:11.000Z",
"updatedAt": "2020-11-02T18:29:04.000Z"
}
],
"count": 11
}
}
-------------------------------------------------------
I1104 15:53:51.261839 12397 resources.go:71] Listing all Ocean resources
I1104 15:53:51.262133 12397 spotinst.go:80] SPOTINST: Request "GET https://api.spotinst.io/ocean/aws/k8s/cluster?accountId=act-*******" details:
---[ REQUEST ]---------------------------------------
GET /ocean/aws/k8s/cluster?accountId=act-******* HTTP/1.1
Host: api.spotinst.io
User-Agent: kubernetes-kops/1.18.2 spotinst-sdk-go/1.49.0 (go1.13.9; linux; amd64)
Accept: application/json
Authorization: Bearer *****
Content-Type: application/json
Accept-Encoding: gzip
-----------------------------------------------------
I1104 15:53:51.459206 12397 spotinst.go:80] SPOTINST: Response "GET https://api.spotinst.io/ocean/aws/k8s/cluster?accountId=act-*******" details:
---[ RESPONSE ]----------------------------------------
HTTP/1.1 200 OK
Content-Length: 352
Access-Control-Allow-Headers: Origin,Accept,Content-Type,Authorization,X-CSRF-Token
Access-Control-Allow-Methods: GET,POST,DELETE,PUT
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Wed, 04 Nov 2020 13:53:51 GMT
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Ratelimit-Limit: 300
X-Ratelimit-Remaining: 299
X-Request-Id: 58d098e7-5826-4b36-ad69-a552db15293f
X-Response-Time: 43ms
{
"request": {
"id": "58d098e7-5826-4b36-ad69-a552db15293f",
"url": "/ocean/aws/k8s/cluster?accountId=act-*******",
"method": "GET",
"timestamp": "2020-11-04T13:53:51.371Z"
},
"response": {
"status": {
"code": 200,
"message": "OK"
},
"kind": "spotinst:ocean:aws:k8s",
"items": [],
"count": 0
}
}
-------------------------------------------------------
I1104 15:53:51.459325 12397 resources.go:405] Found group with corresponding instance group: "nodes.test.example.com"
I1104 15:53:51.459347 12397 resources.go:295] Attempting to fetch all instances of instance group: "nodes.test.example.com" (id: "sig-********")
I1104 15:53:51.459645 12397 spotinst.go:80] SPOTINST: Request "GET https://api.spotinst.io/aws/ec2/group/sig-********/status?accountId=act-*******" details:
---[ REQUEST ]---------------------------------------
GET /aws/ec2/group/sig-********/status?accountId=act-******* HTTP/1.1
Host: api.spotinst.io
User-Agent: kubernetes-kops/1.18.2 spotinst-sdk-go/1.49.0 (go1.13.9; linux; amd64)
Accept: application/json
Authorization: Bearer ****
Content-Type: application/json
Accept-Encoding: gzip
-----------------------------------------------------
I1104 15:53:51.655082 12397 spotinst.go:80] SPOTINST: Response "GET https://api.spotinst.io/aws/ec2/group/sig-********/status?accountId=act-*******" details:
---[ RESPONSE ]----------------------------------------
HTTP/1.1 200 OK
Content-Length: 3146
Access-Control-Allow-Headers: Origin,Accept,Content-Type,Authorization,X-CSRF-Token
Access-Control-Allow-Methods: GET,POST,DELETE,PUT
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Wed, 04 Nov 2020 13:53:51 GMT
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Ratelimit-Limit: 300
X-Ratelimit-Remaining: 299
X-Request-Id: ea120c35-8fcc-427b-a48f-956ad3db7307
X-Response-Time: 40ms
{
"request": {
"id": "ea120c35-8fcc-427b-a48f-956ad3db7307",
"url": "/aws/ec2/group/sig-********/status",
"method": "GET",
"timestamp": "2020-11-04T13:53:51.566Z"
},
"response": {
"status": {
"code": 200,
"message": "OK"
},
"kind": "spotinst:aws:ec2:group",
"items": [
{
"spotInstanceRequestId": "sir-*****",
"instanceId": "i-0ffadee0802927c01",
"instanceType": "t3a.2xlarge",
"product": "Linux/UNIX",
"groupId": "sig-********",
"availabilityZone": "us-east-1f",
"privateIp": "10.40.59.224",
"createdAt": "2020-11-04T13:17:13.000Z",
"publicIp": null,
"status": "fulfilled"
},
{
"spotInstanceRequestId": "sir-*****",
"instanceId": "i-036e06b204f56f316",
"instanceType": "t3a.xlarge",
"product": "Linux/UNIX",
"groupId": "sig-********",
"availabilityZone": "us-east-1f",
"privateIp": "10.40.56.27",
"createdAt": "2020-11-04T12:40:12.000Z",
"publicIp": null,
"status": "fulfilled"
},
{
"spotInstanceRequestId": "sir-*****",
"instanceId": "i-0e4fb331463f91e5b",
"instanceType": "r5.large",
"product": "Linux/UNIX",
"groupId": "sig-********",
"availabilityZone": "us-east-1f",
"privateIp": "10.40.51.35",
"createdAt": "2020-11-03T17:41:47.000Z",
"publicIp": null,
"status": "fulfilled"
},
{
"spotInstanceRequestId": "sir-*****",
"instanceId": "i-091d1b1a0cf8a3486",
"instanceType": "c5.xlarge",
"product": "Linux/UNIX",
"groupId": "sig-********",
"availabilityZone": "us-east-1f",
"privateIp": "10.40.45.241",
"createdAt": "2020-11-04T12:05:15.000Z",
"publicIp": null,
"status": "fulfilled"
},
{
"spotInstanceRequestId": "sir-*****",
"instanceId": "i-0b738537410c47ce9",
"instanceType": "m5a.xlarge",
"product": "Linux/UNIX",
"groupId": "sig-********",
"availabilityZone": "us-east-1f",
"privateIp": "10.40.42.110",
"createdAt": "2020-11-04T12:48:16.000Z",
"publicIp": null,
"status": "fulfilled"
},
{
"spotInstanceRequestId": "sir-*****",
"instanceId": "i-07759285ebe2a969d",
"instanceType": "t3.xlarge",
"product": "Linux/UNIX",
"groupId": "sig-********",
"availabilityZone": "us-east-1f",
"privateIp": "10.40.55.41",
"createdAt": "2020-11-04T12:44:13.000Z",
"publicIp": null,
"status": "fulfilled"
},
{
"spotInstanceRequestId": "sir-*****",
"instanceId": "i-0deead86f7d952607",
"instanceType": "t3a.medium",
"product": "Linux/UNIX",
"groupId": "sig-********",
"availabilityZone": "us-east-1f",
"privateIp": "10.40.57.245",
"createdAt": "2020-11-04T12:01:29.000Z",
"publicIp": null,
"status": "fulfilled"
}
],
"count": 7
}
}
-------------------------------------------------------
I1104 15:53:51.655542 12397 resources.go:377] Adding instance "i-0ffadee0802927c01" (created at: 2020-11-04T13:17:13Z) to instance group: "nodes.test.example.com" (updated at: 2020-11-04T13:15:13Z)
I1104 15:53:51.655577 12397 resources.go:377] Adding instance "i-036e06b204f56f316" (created at: 2020-11-04T12:40:12Z) to instance group: "nodes.test.example.com" (updated at: 2020-11-04T13:15:13Z)
I1104 15:53:51.655596 12397 resources.go:377] Adding instance "i-0e4fb331463f91e5b" (created at: 2020-11-03T17:41:47Z) to instance group: "nodes.test.example.com" (updated at: 2020-11-04T13:15:13Z)
I1104 15:53:51.655612 12397 resources.go:377] Adding instance "i-091d1b1a0cf8a3486" (created at: 2020-11-04T12:05:15Z) to instance group: "nodes.test.example.com" (updated at: 2020-11-04T13:15:13Z)
I1104 15:53:51.655628 12397 resources.go:377] Adding instance "i-0b738537410c47ce9" (created at: 2020-11-04T12:48:16Z) to instance group: "nodes.test.example.com" (updated at: 2020-11-04T13:15:13Z)
I1104 15:53:51.655644 12397 resources.go:377] Adding instance "i-07759285ebe2a969d" (created at: 2020-11-04T12:44:13Z) to instance group: "nodes.test.example.com" (updated at: 2020-11-04T13:15:13Z)
I1104 15:53:51.655660 12397 resources.go:377] Adding instance "i-0deead86f7d952607" (created at: 2020-11-04T12:01:29Z) to instance group: "nodes.test.example.com" (updated at: 2020-11-04T13:15:13Z)
NAME STATUS NEEDUPDATE READY MIN TARGET MAX NODES
nodes NeedsUpdate 6 1 0 0 17 7
Must specify --yes to rolling-update.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment