- ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@n7.pufty.ci.centos.org 'cd /opt/build/go/src/github.com/ceph/ceph-csi && make run-e2e'
Warning: Permanently added 'n7.pufty.ci.centos.org,172.19.3.71' (ECDSA) to the list of known hosts.
cephcsi image settings: quay.io/cephcsi/cephcsi version canary
cd e2e &&
../e2e.test -test.v -test.timeout="90m" --deploy-timeout="10" --cephcsi-namespace=cephcsi-e2e-7ebc2ac5 ../e2e.test: /lib64/librados.so.2: no version information available (required by ../e2e.test) Feb 11 11:48:18.761: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used. timeout for deploytimeout 10 === RUN TestE2E Running Suite: E2e Suite ======================== Random Seed: �[1m1613044098�[0m - Will randomize all specs Will run �[1m4�[0m of �[1m4�[0m specs
I0211 11:48:18.761315 75146 test_context.go:457] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready �[0mRBD Upgrade Testing�[0m �[90mTest RBD CSI�[0m �[1mTest RBD CSI�[0m �[37m/go/src/github.com/ceph/ceph-csi/e2e/upgrade-rbd.go:141�[0m [BeforeEach] RBD Upgrade Testing /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:174 �[1mSTEP�[0m: Creating a kubernetes client Feb 11 11:48:18.761: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename upgrade-test-rbd Feb 11 11:48:18.814: INFO: Found PodSecurityPolicies; testing pod creation to see if PodSecurityPolicy is enabled Feb 11 11:48:18.824: INFO: PSP annotation exists on dry run pod: "csi-snapshotter-psp"; assuming PodSecurityPolicy is enabled Feb 11 11:48:18.844: INFO: Found ClusterRoles; assuming RBAC is enabled. �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in upgrade-test-rbd-8081 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] RBD Upgrade Testing /go/src/github.com/ceph/ceph-csi/e2e/upgrade-rbd.go:37 [AfterEach] RBD Upgrade Testing /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:175 Feb 11 11:48:18.959: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "upgrade-test-rbd-8081" for this suite. [AfterEach] RBD Upgrade Testing /go/src/github.com/ceph/ceph-csi/e2e/upgrade-rbd.go:88
�[36m�[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.207 seconds]�[0m RBD Upgrade Testing �[90m/go/src/github.com/ceph/ceph-csi/e2e/upgrade-rbd.go:19�[0m �[36m�[1mTest RBD CSI [BeforeEach]�[0m �[90m/go/src/github.com/ceph/ceph-csi/e2e/upgrade-rbd.go:140�[0m Test RBD CSI �[90m/go/src/github.com/ceph/ceph-csi/e2e/upgrade-rbd.go:141�[0m
�[36mSkipping RBD Upgrade Testing�[0m
/go/src/github.com/ceph/ceph-csi/e2e/upgrade-rbd.go:39
�[90m------------------------------�[0m �[0mRBD�[0m �[90mTest RBD CSI�[0m �[1mTest RBD CSI�[0m �[37m/go/src/github.com/ceph/ceph-csi/e2e/rbd.go:253�[0m [BeforeEach] RBD /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:174 �[1mSTEP�[0m: Creating a kubernetes client Feb 11 11:48:18.969: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename rbd �[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in rbd-1318 �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [BeforeEach] RBD /go/src/github.com/ceph/ceph-csi/e2e/rbd.go:157 Feb 11 11:48:19.143: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --ignore-not-found=true --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:48:19.322: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:48:19.322: INFO: stdout: "clusterrole.rbac.authorization.k8s.io "rbd-external-provisioner-runner" deleted\nclusterrolebinding.rbac.authorization.k8s.io "rbd-csi-provisioner-role" deleted\n" Feb 11 11:48:19.323: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 delete --ignore-not-found=true --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:19.479: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:48:19.479: INFO: stdout: "clusterrole.rbac.authorization.k8s.io "rbd-csi-nodeplugin" deleted\nclusterrolebinding.rbac.authorization.k8s.io "rbd-csi-nodeplugin" deleted\n" Feb 11 11:48:19.480: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:20.501: INFO: stderr: "" Feb 11 11:48:20.501: INFO: stdout: "service/csi-rbdplugin-provisioner created\ndeployment.apps/csi-rbdplugin-provisioner created\n" Feb 11 11:48:20.501: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:21.036: INFO: stderr: "" Feb 11 11:48:21.036: INFO: stdout: "serviceaccount/rbd-csi-provisioner created\nclusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created\nclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created\nrole.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created\nrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created\n" Feb 11 11:48:21.036: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create -f -' Feb 11 11:48:21.423: INFO: stderr: "" Feb 11 11:48:21.423: INFO: stdout: "podsecuritypolicy.policy/rbd-csi-provisioner-psp created\nrole.rbac.authorization.k8s.io/rbd-csi-provisioner-psp created\nrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-psp created\n" Feb 11 11:48:21.423: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:21.772: INFO: stderr: "" Feb 11 11:48:21.772: INFO: stdout: "daemonset.apps/csi-rbdplugin created\nservice/csi-metrics-rbdplugin created\n" Feb 11 11:48:21.772: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:22.166: INFO: stderr: "" Feb 11 11:48:22.166: INFO: stdout: "serviceaccount/rbd-csi-nodeplugin created\nclusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created\nclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created\n" Feb 11 11:48:22.166: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:22.505: INFO: stderr: "" Feb 11 11:48:22.505: INFO: stdout: "podsecuritypolicy.policy/rbd-csi-nodeplugin-psp created\nrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin-psp created\nrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-psp created\n" Feb 11 11:48:22.516: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:48:22.516: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:48:24.921: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:48:24.921: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:48:27.706: INFO: ExecWithOptions {Command:[/bin/sh -c ceph auth get-key client.admin] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:48:27.706: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:48:29.878: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 delete cm ceph-csi-encryption-kms-config --namespace cephcsi-e2e-7ebc2ac5 --ignore-not-found=true' Feb 11 11:48:30.011: INFO: stderr: "" Feb 11 11:48:30.011: INFO: stdout: "" Feb 11 11:48:30.011: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:30.710: INFO: stderr: "" Feb 11 11:48:30.710: INFO: stdout: "service/vault created\ndeployment.apps/vault created\nconfigmap/init-scripts created\njob.batch/vault-init-job created\n" Feb 11 11:48:30.711: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:31.058: INFO: stderr: "" Feb 11 11:48:31.058: INFO: stdout: "serviceaccount/rbd-csi-vault-token-review created\nclusterrole.rbac.authorization.k8s.io/rbd-csi-vault-token-review created\nclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-vault-token-review created\n" Feb 11 11:48:31.058: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:31.403: INFO: stderr: "" Feb 11 11:48:31.403: INFO: stdout: "configmap/ceph-csi-encryption-kms-config created\n" Feb 11 11:48:31.403: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 create --namespace=cephcsi-e2e-7ebc2ac5 -f -' Feb 11 11:48:31.841: INFO: stderr: "" Feb 11 11:48:31.842: INFO: stdout: "podsecuritypolicy.policy/rbd-csi-vault-token-review-psp created\nrole.rbac.authorization.k8s.io/rbd-csi-vault-token-review-psp created\nrolebinding.rbac.authorization.k8s.io/rbd-csi-vault-token-review-psp created\n" Feb 11 11:48:31.846: INFO: Waiting up to vault-64df5bfc9b-55f9t to be in Running state Feb 11 11:48:31.849: INFO: vault-64df5bfc9b-55f9t app is in Pending phase expected to be in Running state (0 seconds elapsed) [It] Test RBD CSI /go/src/github.com/ceph/ceph-csi/e2e/rbd.go:253 �[1mSTEP�[0m: checking provisioner deployment is running Feb 11 11:48:33.856: INFO: deployment status: expected replica count 1 running replica count 0 Feb 11 11:48:35.860: INFO: deployment status: expected replica count 1 running replica count 0 Feb 11 11:48:37.860: INFO: deployment status: expected replica count 1 running replica count 0 Feb 11 11:48:39.861: INFO: deployment status: expected replica count 1 running replica count 0 Feb 11 11:48:41.860: INFO: deployment status: expected replica count 1 running replica count 0 �[1mSTEP�[0m: checking nodeplugin deamonset pods are running Feb 11 11:48:43.860: INFO: Waiting up to 10m0s for all daemonsets in namespace 'cephcsi-e2e-7ebc2ac5' to start Feb 11 11:48:43.863: INFO: 1 / 1 pods ready in namespace 'cephcsi-e2e-7ebc2ac5' in daemonset 'csi-rbdplugin' (0 seconds elapsed) �[1mSTEP�[0m: create a PVC and validate owner Feb 11 11:48:43.870: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-1318 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:48:43.871: INFO: waiting for PVC rbd-1318 (0 seconds elapsed) Feb 11 11:48:45.878: INFO: waiting for PVC rbd-1318 (2 seconds elapsed) Feb 11 11:48:47.878: INFO: waiting for PVC rbd-1318 (4 seconds elapsed) Feb 11 11:48:47.883: INFO: Waiting for PV pvc-d1a94f14-293f-4034-92e7-9086f1da9138 to bind to PVC rbd-1318 Feb 11 11:48:47.883: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-1318] to have phase Bound Feb 11 11:48:47.886: INFO: PersistentVolumeClaim rbd-1318 found and phase=Bound (2.521824ms) Feb 11 11:48:47.886: INFO: Waiting up to 3m0s for PersistentVolume pvc-d1a94f14-293f-4034-92e7-9086f1da9138 to have phase Bound Feb 11 11:48:47.888: INFO: PersistentVolume pvc-d1a94f14-293f-4034-92e7-9086f1da9138 found and phase=Bound (2.371537ms) Feb 11 11:48:47.904: INFO: ExecWithOptions {Command:[/bin/sh -c rados --pool=replicapool getomapval csi.volume.17a2cb45-6c5f-11eb-a35b-f20347e291c7 csi.volume.owner] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:48:47.904: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:48:48.883: INFO: found image journal csi.volume.17a2cb45-6c5f-11eb-a35b-f20347e291c7 in pool replicapool Feb 11 11:48:48.883: INFO: Deleting PersistentVolumeClaim rbd-1318 on namespace rbd-1318 Feb 11 11:48:48.896: INFO: waiting for PVC rbd-1318 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:48:50.906: INFO: waiting for PVC rbd-1318 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:48:50.918: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:48:50.918: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: create a PVC and bind it to an app Feb 11 11:48:51.801: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:48:51.801: INFO: waiting for PVC rbd-pvc (0 seconds elapsed) Feb 11 11:48:53.807: INFO: waiting for PVC rbd-pvc (2 seconds elapsed) Feb 11 11:48:53.814: INFO: Waiting for PV pvc-e491e625-ef4d-44e9-b253-94f0d54a16b1 to bind to PVC rbd-pvc Feb 11 11:48:53.814: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound Feb 11 11:48:53.816: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (2.312214ms) Feb 11 11:48:53.816: INFO: Waiting up to 3m0s for PersistentVolume pvc-e491e625-ef4d-44e9-b253-94f0d54a16b1 to have phase Bound Feb 11 11:48:53.818: INFO: PersistentVolume pvc-e491e625-ef4d-44e9-b253-94f0d54a16b1 found and phase=Bound (2.41473ms) Feb 11 11:48:53.830: INFO: Waiting up to csi-rbd-demo-pod to be in Running state Feb 11 11:48:53.838: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:48:55.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:48:57.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:48:59.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:49:01.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:49:03.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:49:05.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:49:07.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:49:09.842: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:49:11.841: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:49:13.849: INFO: Waiting for pod csi-rbd-demo-pod to be deleted Feb 11 11:49:13.852: INFO: csi-rbd-demo-pod app to be deleted (0 seconds elapsed) Feb 11 11:49:15.857: INFO: csi-rbd-demo-pod app to be deleted (2 seconds elapsed) Feb 11 11:49:17.856: INFO: Deleting PersistentVolumeClaim rbd-pvc on namespace rbd-1318 Feb 11 11:49:17.866: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:49:19.873: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:49:19.887: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:49:19.887: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: create a PVC and bind it to an app with normal user Feb 11 11:49:20.843: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-1318 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:49:20.843: INFO: waiting for PVC rbd-1318 (0 seconds elapsed) Feb 11 11:49:22.848: INFO: waiting for PVC rbd-1318 (2 seconds elapsed) Feb 11 11:49:22.853: INFO: Waiting for PV pvc-4bf6fff5-bcc0-484d-b23c-3542524da2fb to bind to PVC rbd-1318 Feb 11 11:49:22.853: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-1318] to have phase Bound Feb 11 11:49:22.855: INFO: PersistentVolumeClaim rbd-1318 found and phase=Bound (2.259122ms) Feb 11 11:49:22.855: INFO: Waiting up to 3m0s for PersistentVolume pvc-4bf6fff5-bcc0-484d-b23c-3542524da2fb to have phase Bound Feb 11 11:49:22.858: INFO: PersistentVolume pvc-4bf6fff5-bcc0-484d-b23c-3542524da2fb found and phase=Bound (2.158664ms) Feb 11 11:49:22.876: INFO: Waiting up to pod-run-as-non-root to be in Running state Feb 11 11:49:22.885: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:49:24.890: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:49:26.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:49:28.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:49:30.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:49:32.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:49:34.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:49:36.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:49:38.890: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:49:40.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:49:42.890: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:49:44.891: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:49:46.890: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:49:48.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:49:50.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (28 seconds elapsed) Feb 11 11:49:53.102: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (30 seconds elapsed) Feb 11 11:49:54.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (32 seconds elapsed) Feb 11 11:49:56.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (34 seconds elapsed) Feb 11 11:49:58.889: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (36 seconds elapsed) Feb 11 11:50:08.064: INFO: pod-run-as-non-root app is in Pending phase expected to be in Running state (45 seconds elapsed) Feb 11 11:50:08.893: INFO: ExecWithOptions {Command:[/bin/sh -c echo testing > /target/testing] Namespace:rbd-1318 PodName:pod-run-as-non-root ContainerName:write-pod Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:50:08.893: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:50:09.116: INFO: Waiting for pod pod-run-as-non-root to be deleted Feb 11 11:50:09.120: INFO: pod-run-as-non-root app to be deleted (0 seconds elapsed) Feb 11 11:50:11.124: INFO: pod-run-as-non-root app to be deleted (2 seconds elapsed) Feb 11 11:50:13.125: INFO: pod-run-as-non-root app to be deleted (4 seconds elapsed) Feb 11 11:50:15.124: INFO: pod-run-as-non-root app to be deleted (6 seconds elapsed) Feb 11 11:50:17.124: INFO: pod-run-as-non-root app to be deleted (8 seconds elapsed) Feb 11 11:50:19.124: INFO: pod-run-as-non-root app to be deleted (10 seconds elapsed) Feb 11 11:50:21.123: INFO: pod-run-as-non-root app to be deleted (12 seconds elapsed) Feb 11 11:50:23.124: INFO: pod-run-as-non-root app to be deleted (14 seconds elapsed) Feb 11 11:50:25.124: INFO: pod-run-as-non-root app to be deleted (16 seconds elapsed) Feb 11 11:50:27.124: INFO: pod-run-as-non-root app to be deleted (18 seconds elapsed) Feb 11 11:50:29.124: INFO: pod-run-as-non-root app to be deleted (20 seconds elapsed) Feb 11 11:50:31.124: INFO: pod-run-as-non-root app to be deleted (22 seconds elapsed) Feb 11 11:50:33.124: INFO: pod-run-as-non-root app to be deleted (24 seconds elapsed) Feb 11 11:50:35.125: INFO: pod-run-as-non-root app to be deleted (26 seconds elapsed) Feb 11 11:50:37.124: INFO: pod-run-as-non-root app to be deleted (28 seconds elapsed) Feb 11 11:50:39.124: INFO: pod-run-as-non-root app to be deleted (30 seconds elapsed) Feb 11 11:50:41.125: INFO: pod-run-as-non-root app to be deleted (32 seconds elapsed) Feb 11 11:50:43.124: INFO: pod-run-as-non-root app to be deleted (34 seconds elapsed) Feb 11 11:50:45.124: INFO: pod-run-as-non-root app to be deleted (36 seconds elapsed) Feb 11 11:50:47.124: INFO: pod-run-as-non-root app to be deleted (38 seconds elapsed) Feb 11 11:50:49.125: INFO: Deleting PersistentVolumeClaim rbd-1318 on namespace rbd-1318 Feb 11 11:50:49.136: INFO: waiting for PVC rbd-1318 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:50:51.139: INFO: waiting for PVC rbd-1318 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:50:51.152: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:50:51.152: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: create a PVC and bind it to an app with ext4 as the FS Feb 11 11:50:52.028: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:50:52.177: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:50:52.177: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:50:52.184: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:50:52.184: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:50:54.398: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:50:54.399: INFO: waiting for PVC rbd-pvc (0 seconds elapsed) Feb 11 11:50:56.406: INFO: waiting for PVC rbd-pvc (2 seconds elapsed) Feb 11 11:50:56.412: INFO: Waiting for PV pvc-c4a96bc6-6cfb-4510-8b46-a8337f850b5d to bind to PVC rbd-pvc Feb 11 11:50:56.412: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound Feb 11 11:50:56.414: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (2.308488ms) Feb 11 11:50:56.414: INFO: Waiting up to 3m0s for PersistentVolume pvc-c4a96bc6-6cfb-4510-8b46-a8337f850b5d to have phase Bound Feb 11 11:50:56.417: INFO: PersistentVolume pvc-c4a96bc6-6cfb-4510-8b46-a8337f850b5d found and phase=Bound (2.412405ms) Feb 11 11:50:56.428: INFO: Waiting up to csi-rbd-demo-pod to be in Running state Feb 11 11:50:56.433: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:50:58.437: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:51:00.436: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:51:02.436: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:51:04.436: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:51:06.436: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:51:08.441: INFO: Waiting for pod csi-rbd-demo-pod to be deleted Feb 11 11:51:08.446: INFO: csi-rbd-demo-pod app to be deleted (0 seconds elapsed) Feb 11 11:51:10.450: INFO: csi-rbd-demo-pod app to be deleted (2 seconds elapsed) Feb 11 11:51:12.450: INFO: csi-rbd-demo-pod app to be deleted (4 seconds elapsed) Feb 11 11:51:14.450: INFO: csi-rbd-demo-pod app to be deleted (6 seconds elapsed) Feb 11 11:51:16.450: INFO: csi-rbd-demo-pod app to be deleted (8 seconds elapsed) Feb 11 11:51:18.454: INFO: Deleting PersistentVolumeClaim rbd-pvc on namespace rbd-1318 Feb 11 11:51:18.478: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:51:20.483: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:51:20.497: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:51:20.497: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:51:21.388: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:51:21.527: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:51:21.527: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:51:21.538: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:51:21.538: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: create a PVC and bind it to an app with encrypted RBD volume Feb 11 11:51:23.624: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:51:23.779: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:51:23.779: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:51:23.787: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:51:23.787: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:51:26.016: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:51:26.016: INFO: waiting for PVC rbd-pvc (0 seconds elapsed) Feb 11 11:51:28.021: INFO: waiting for PVC rbd-pvc (2 seconds elapsed) Feb 11 11:51:28.027: INFO: Waiting for PV pvc-5585a73e-9507-4ea1-a888-911032f091ed to bind to PVC rbd-pvc Feb 11 11:51:28.027: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound Feb 11 11:51:28.030: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (2.500424ms) Feb 11 11:51:28.030: INFO: Waiting up to 3m0s for PersistentVolume pvc-5585a73e-9507-4ea1-a888-911032f091ed to have phase Bound Feb 11 11:51:28.032: INFO: PersistentVolume pvc-5585a73e-9507-4ea1-a888-911032f091ed found and phase=Bound (2.314599ms) Feb 11 11:51:28.046: INFO: Waiting up to csi-rbd-demo-pod to be in Running state Feb 11 11:51:28.051: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:51:30.055: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:51:32.054: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:51:34.054: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:51:36.055: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:51:38.055: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:51:40.055: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:51:42.056: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:51:44.055: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:51:46.054: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:51:48.067: INFO: ExecWithOptions {Command:[/bin/sh -c rbd image-meta get replicapool/csi-vol-7842c14a-6c5f-11eb-a35b-f20347e291c7 .rbd.csi.ceph.com/encrypted] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:51:48.067: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:51:49.033: INFO: ExecWithOptions {Command:[/bin/sh -c lsblk -o TYPE,MOUNTPOINT | grep '/var/lib/www/html' | awk '{print $1}'] Namespace:rbd-1318 PodName:csi-rbd-demo-pod ContainerName:web-server Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:51:49.033: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:51:49.258: INFO: Waiting for pod csi-rbd-demo-pod to be deleted Feb 11 11:51:49.262: INFO: csi-rbd-demo-pod app to be deleted (0 seconds elapsed) Feb 11 11:51:51.266: INFO: csi-rbd-demo-pod app to be deleted (2 seconds elapsed) Feb 11 11:51:53.266: INFO: csi-rbd-demo-pod app to be deleted (4 seconds elapsed) Feb 11 11:51:55.266: INFO: csi-rbd-demo-pod app to be deleted (6 seconds elapsed) Feb 11 11:51:57.266: INFO: csi-rbd-demo-pod app to be deleted (8 seconds elapsed) Feb 11 11:51:59.266: INFO: Deleting PersistentVolumeClaim rbd-pvc on namespace rbd-1318 Feb 11 11:51:59.278: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:52:01.283: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:52:01.296: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:01.296: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:02.188: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:52:02.325: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:52:02.325: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:52:02.332: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:02.332: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: create a PVC and bind it to an app with encrypted RBD volume with VaultKMS Feb 11 11:52:04.501: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:52:04.654: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:52:04.654: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:52:04.661: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:04.661: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:06.836: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:52:06.837: INFO: waiting for PVC rbd-pvc (0 seconds elapsed) Feb 11 11:52:08.841: INFO: waiting for PVC rbd-pvc (2 seconds elapsed) Feb 11 11:52:08.847: INFO: Waiting for PV pvc-91771f38-b09c-4e1f-ae57-4f7d35320797 to bind to PVC rbd-pvc Feb 11 11:52:08.847: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound Feb 11 11:52:08.849: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (2.018981ms) Feb 11 11:52:08.849: INFO: Waiting up to 3m0s for PersistentVolume pvc-91771f38-b09c-4e1f-ae57-4f7d35320797 to have phase Bound Feb 11 11:52:08.851: INFO: PersistentVolume pvc-91771f38-b09c-4e1f-ae57-4f7d35320797 found and phase=Bound (2.055592ms) Feb 11 11:52:08.863: INFO: Waiting up to csi-rbd-demo-pod to be in Running state Feb 11 11:52:08.868: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:52:10.872: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:52:12.871: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:52:14.872: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:52:16.871: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:52:18.879: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:52:20.875: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:52:22.872: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:52:24.872: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:52:26.871: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:52:28.871: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:52:30.882: INFO: ExecWithOptions {Command:[/bin/sh -c rbd image-meta get replicapool/csi-vol-909ade38-6c5f-11eb-a35b-f20347e291c7 .rbd.csi.ceph.com/encrypted] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:30.882: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:31.828: INFO: ExecWithOptions {Command:[/bin/sh -c lsblk -o TYPE,MOUNTPOINT | grep '/var/lib/www/html' | awk '{print $1}'] Namespace:rbd-1318 PodName:csi-rbd-demo-pod ContainerName:web-server Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:31.828: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:32.023: INFO: ExecWithOptions {Command:[/bin/sh -c vault login -address=http://vault.cephcsi-e2e-7ebc2ac5.svc.cluster.local:8200 sample_root_token_id > /dev/null && vault kv get -address=http://vault.cephcsi-e2e-7ebc2ac5.svc.cluster.local:8200 -field=data /secret/ceph-csi/0001-0024-316ee4fa-2016-47ee-98cb-dff247fba6cc-0000000000000002-909ade38-6c5f-11eb-a35b-f20347e291c7] Namespace:cephcsi-e2e-7ebc2ac5 PodName:vault-64df5bfc9b-55f9t ContainerName:vault Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:32.023: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:32.397: INFO: Waiting for pod csi-rbd-demo-pod to be deleted Feb 11 11:52:32.404: INFO: csi-rbd-demo-pod app to be deleted (0 seconds elapsed) Feb 11 11:52:34.408: INFO: csi-rbd-demo-pod app to be deleted (2 seconds elapsed) Feb 11 11:52:36.407: INFO: Deleting PersistentVolumeClaim rbd-pvc on namespace rbd-1318 Feb 11 11:52:36.422: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:52:38.425: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:52:38.437: INFO: ExecWithOptions {Command:[/bin/sh -c vault login -address=http://vault.cephcsi-e2e-7ebc2ac5.svc.cluster.local:8200 sample_root_token_id > /dev/null && vault kv get -address=http://vault.cephcsi-e2e-7ebc2ac5.svc.cluster.local:8200 -field=data /secret/ceph-csi/0001-0024-316ee4fa-2016-47ee-98cb-dff247fba6cc-0000000000000002-909ade38-6c5f-11eb-a35b-f20347e291c7] Namespace:cephcsi-e2e-7ebc2ac5 PodName:vault-64df5bfc9b-55f9t ContainerName:vault Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:38.437: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:38.834: INFO: command vault login -address=http://vault.cephcsi-e2e-7ebc2ac5.svc.cluster.local:8200 sample_root_token_id > /dev/null && vault kv get -address=http://vault.cephcsi-e2e-7ebc2ac5.svc.cluster.local:8200 -field=data /secret/ceph-csi/0001-0024-316ee4fa-2016-47ee-98cb-dff247fba6cc-0000000000000002-909ade38-6c5f-11eb-a35b-f20347e291c7 failed: command terminated with exit code 2 Feb 11 11:52:38.843: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:38.843: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:39.778: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:52:39.916: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:52:39.916: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:52:39.923: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:39.923: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: create a PVC and bind it to an app with encrypted RBD volume with VaultTokensKMS Feb 11 11:52:42.038: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:52:42.175: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:52:42.175: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:52:42.181: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:52:42.181: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:52:44.327: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:52:44.327: INFO: waiting for PVC rbd-pvc (0 seconds elapsed) Feb 11 11:52:46.334: INFO: waiting for PVC rbd-pvc (2 seconds elapsed) Feb 11 11:52:46.341: INFO: Waiting for PV pvc-78078ca1-4fe7-44ec-bff6-1da762b789be to bind to PVC rbd-pvc Feb 11 11:52:46.341: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound Feb 11 11:52:46.344: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (3.502545ms) Feb 11 11:52:46.344: INFO: Waiting up to 3m0s for PersistentVolume pvc-78078ca1-4fe7-44ec-bff6-1da762b789be to have phase Bound Feb 11 11:52:46.347: INFO: PersistentVolume pvc-78078ca1-4fe7-44ec-bff6-1da762b789be found and phase=Bound (2.395199ms) Feb 11 11:52:46.359: INFO: Waiting up to csi-rbd-demo-pod to be in Running state Feb 11 11:52:46.364: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:52:48.368: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:52:50.368: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:52:52.368: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:52:54.368: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:52:56.367: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:52:58.369: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:53:00.367: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:53:02.367: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:53:04.369: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:53:06.368: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:53:08.369: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:53:10.368: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:53:12.368: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:53:14.367: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (28 seconds elapsed) Feb 11 11:53:16.378: INFO: ExecWithOptions {Command:[/bin/sh -c rbd image-meta get replicapool/csi-vol-a6fde383-6c5f-11eb-a35b-f20347e291c7 .rbd.csi.ceph.com/encrypted] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:53:16.378: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:17.335: INFO: ExecWithOptions {Command:[/bin/sh -c lsblk -o TYPE,MOUNTPOINT | grep '/var/lib/www/html' | awk '{print $1}'] Namespace:rbd-1318 PodName:csi-rbd-demo-pod ContainerName:web-server Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:53:17.335: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:17.540: INFO: Waiting for pod csi-rbd-demo-pod to be deleted Feb 11 11:53:17.544: INFO: csi-rbd-demo-pod app to be deleted (0 seconds elapsed) Feb 11 11:53:19.547: INFO: csi-rbd-demo-pod app to be deleted (2 seconds elapsed) Feb 11 11:53:21.547: INFO: csi-rbd-demo-pod app to be deleted (4 seconds elapsed) Feb 11 11:53:23.547: INFO: csi-rbd-demo-pod app to be deleted (6 seconds elapsed) Feb 11 11:53:25.548: INFO: csi-rbd-demo-pod app to be deleted (8 seconds elapsed) Feb 11 11:53:27.547: INFO: csi-rbd-demo-pod app to be deleted (10 seconds elapsed) Feb 11 11:53:29.547: INFO: Deleting PersistentVolumeClaim rbd-pvc on namespace rbd-1318 Feb 11 11:53:29.556: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:53:31.563: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:53:31.576: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:53:31.576: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:32.456: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.176:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-7ebc2ac5 --namespace=cephcsi-e2e-7ebc2ac5 delete -f -' Feb 11 11:53:32.603: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Feb 11 11:53:32.603: INFO: stdout: "storageclass.storage.k8s.io "csi-rbd-sc" deleted\n" Feb 11 11:53:32.609: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:53:32.609: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: create a PVC clone and bind it to an app Feb 11 11:53:34.858: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:53:34.858: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:37.007: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:37.029: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:53:37.029: INFO: waiting for PVC rbd-pvc (0 seconds elapsed) Feb 11 11:53:39.034: INFO: waiting for PVC rbd-pvc (2 seconds elapsed) Feb 11 11:53:39.040: INFO: Waiting for PV pvc-de0bb48e-fa5c-4208-ad8b-9543fbf96dce to bind to PVC rbd-pvc Feb 11 11:53:39.040: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound Feb 11 11:53:39.042: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (2.79614ms) Feb 11 11:53:39.042: INFO: Waiting up to 3m0s for PersistentVolume pvc-de0bb48e-fa5c-4208-ad8b-9543fbf96dce to have phase Bound Feb 11 11:53:39.045: INFO: PersistentVolume pvc-de0bb48e-fa5c-4208-ad8b-9543fbf96dce found and phase=Bound (2.322429ms) Feb 11 11:53:39.056: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:53:39.056: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.980: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.980: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.980: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.980: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.981: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.981: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.980: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.981: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.981: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:39.982: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:53:40.005: INFO: snapshot with name rbd-13187 created in rbd-1318 namespace Feb 11 11:53:40.005: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13187 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.005: INFO: waiting for snapshot rbd-13187 (0 seconds elapsed) Feb 11 11:53:40.005: INFO: snapshot with name rbd-13189 created in rbd-1318 namespace Feb 11 11:53:40.005: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13189 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.005: INFO: waiting for snapshot rbd-13189 (0 seconds elapsed) Feb 11 11:53:40.005: INFO: snapshot with name rbd-13184 created in rbd-1318 namespace Feb 11 11:53:40.005: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13184 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.005: INFO: waiting for snapshot rbd-13184 (0 seconds elapsed) Feb 11 11:53:40.005: INFO: snapshot with name rbd-13180 created in rbd-1318 namespace Feb 11 11:53:40.005: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13180 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.005: INFO: waiting for snapshot rbd-13180 (0 seconds elapsed) Feb 11 11:53:40.006: INFO: snapshot with name rbd-13183 created in rbd-1318 namespace Feb 11 11:53:40.006: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13183 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.006: INFO: waiting for snapshot rbd-13183 (0 seconds elapsed) Feb 11 11:53:40.006: INFO: snapshot with name rbd-13188 created in rbd-1318 namespace Feb 11 11:53:40.006: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13188 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.006: INFO: waiting for snapshot rbd-13188 (0 seconds elapsed) Feb 11 11:53:40.006: INFO: snapshot with name rbd-13185 created in rbd-1318 namespace Feb 11 11:53:40.006: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13185 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.006: INFO: waiting for snapshot rbd-13185 (0 seconds elapsed) Feb 11 11:53:40.007: INFO: snapshot with name rbd-13182 created in rbd-1318 namespace Feb 11 11:53:40.007: INFO: snapshot with name rbd-13186 created in rbd-1318 namespace Feb 11 11:53:40.007: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13182 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.007: INFO: waiting for snapshot rbd-13182 (0 seconds elapsed) Feb 11 11:53:40.007: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13186 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.007: INFO: waiting for snapshot rbd-13186 (0 seconds elapsed) Feb 11 11:53:40.007: INFO: snapshot with name rbd-13181 created in rbd-1318 namespace Feb 11 11:53:40.007: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13181 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be in ready state Feb 11 11:53:40.007: INFO: waiting for snapshot rbd-13181 (0 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13184 (2 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13180 (2 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13187 (2 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13181 (2 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13188 (2 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13186 (2 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13183 (2 seconds elapsed) Feb 11 11:53:42.018: INFO: waiting for snapshot rbd-13185 (2 seconds elapsed) Feb 11 11:53:42.019: INFO: waiting for snapshot rbd-13189 (2 seconds elapsed) Feb 11 11:53:42.019: INFO: waiting for snapshot rbd-13182 (2 seconds elapsed) Feb 11 11:53:44.018: INFO: waiting for snapshot rbd-13184 (4 seconds elapsed) Feb 11 11:53:44.018: INFO: waiting for snapshot rbd-13180 (4 seconds elapsed) Feb 11 11:53:44.018: INFO: waiting for snapshot rbd-13183 (4 seconds elapsed) Feb 11 11:53:44.018: INFO: waiting for snapshot rbd-13188 (4 seconds elapsed) Feb 11 11:53:44.018: INFO: waiting for snapshot rbd-13187 (4 seconds elapsed) Feb 11 11:53:44.018: INFO: waiting for snapshot rbd-13181 (4 seconds elapsed) Feb 11 11:53:44.019: INFO: waiting for snapshot rbd-13189 (4 seconds elapsed) Feb 11 11:53:44.019: INFO: waiting for snapshot rbd-13186 (4 seconds elapsed) Feb 11 11:53:44.019: INFO: waiting for snapshot rbd-13185 (4 seconds elapsed) Feb 11 11:53:44.019: INFO: waiting for snapshot rbd-13182 (4 seconds elapsed) Feb 11 11:53:46.018: INFO: waiting for snapshot rbd-13184 (6 seconds elapsed) Feb 11 11:53:46.018: INFO: waiting for snapshot rbd-13181 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13180 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13182 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13188 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13187 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13186 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13183 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13189 (6 seconds elapsed) Feb 11 11:53:46.019: INFO: waiting for snapshot rbd-13185 (6 seconds elapsed) Feb 11 11:53:46.081: INFO: snapshot rbd-13180 in false state Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13184 (8 seconds elapsed) Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13180 (8 seconds elapsed) Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13181 (8 seconds elapsed) Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13187 (8 seconds elapsed) Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13188 (8 seconds elapsed) Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13183 (8 seconds elapsed) Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13186 (8 seconds elapsed) Feb 11 11:53:48.018: INFO: waiting for snapshot rbd-13185 (8 seconds elapsed) Feb 11 11:53:48.019: INFO: waiting for snapshot rbd-13182 (8 seconds elapsed) Feb 11 11:53:48.019: INFO: waiting for snapshot rbd-13189 (8 seconds elapsed) Feb 11 11:53:48.024: INFO: snapshot rbd-13180 in false state Feb 11 11:53:48.024: INFO: snapshot rbd-13185 in false state Feb 11 11:53:48.027: INFO: snapshot rbd-13181 in false state Feb 11 11:53:48.027: INFO: snapshot rbd-13183 in false state Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13180 (10 seconds elapsed) Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13188 (10 seconds elapsed) Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13181 (10 seconds elapsed) Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13184 (10 seconds elapsed) Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13187 (10 seconds elapsed) Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13183 (10 seconds elapsed) Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13186 (10 seconds elapsed) Feb 11 11:53:50.018: INFO: waiting for snapshot rbd-13185 (10 seconds elapsed) Feb 11 11:53:50.019: INFO: waiting for snapshot rbd-13182 (10 seconds elapsed) Feb 11 11:53:50.019: INFO: waiting for snapshot rbd-13189 (10 seconds elapsed) Feb 11 11:53:50.023: INFO: snapshot rbd-13180 in false state Feb 11 11:53:50.026: INFO: snapshot rbd-13181 in false state Feb 11 11:53:50.027: INFO: snapshot rbd-13187 in false state Feb 11 11:53:50.028: INFO: snapshot rbd-13184 in false state Feb 11 11:53:50.029: INFO: snapshot rbd-13182 in false state Feb 11 11:53:50.034: INFO: snapshot rbd-13183 in false state Feb 11 11:53:50.034: INFO: snapshot rbd-13186 in false state Feb 11 11:53:50.034: INFO: snapshot rbd-13185 in false state Feb 11 11:53:52.018: INFO: waiting for snapshot rbd-13180 (12 seconds elapsed) Feb 11 11:53:52.018: INFO: waiting for snapshot rbd-13184 (12 seconds elapsed) Feb 11 11:53:52.018: INFO: waiting for snapshot rbd-13187 (12 seconds elapsed) Feb 11 11:53:52.018: INFO: waiting for snapshot rbd-13183 (12 seconds elapsed) Feb 11 11:53:52.018: INFO: waiting for snapshot rbd-13186 (12 seconds elapsed) Feb 11 11:53:52.018: INFO: waiting for snapshot rbd-13188 (12 seconds elapsed) Feb 11 11:53:52.018: INFO: waiting for snapshot rbd-13181 (12 seconds elapsed) Feb 11 11:53:52.019: INFO: waiting for snapshot rbd-13185 (12 seconds elapsed) Feb 11 11:53:52.019: INFO: waiting for snapshot rbd-13189 (12 seconds elapsed) Feb 11 11:53:52.019: INFO: waiting for snapshot rbd-13182 (12 seconds elapsed) Feb 11 11:53:52.029: INFO: snapshot rbd-13184 in false state Feb 11 11:53:52.029: INFO: snapshot rbd-13180 in false state Feb 11 11:53:52.030: INFO: snapshot rbd-13181 in false state Feb 11 11:53:52.031: INFO: snapshot rbd-13187 in false state Feb 11 11:53:52.034: INFO: snapshot rbd-13189 in false state Feb 11 11:53:52.037: INFO: snapshot rbd-13186 in false state Feb 11 11:53:52.038: INFO: snapshot rbd-13188 in false state Feb 11 11:53:52.038: INFO: snapshot rbd-13182 in false state Feb 11 11:53:52.038: INFO: snapshot rbd-13185 in false state Feb 11 11:53:52.038: INFO: snapshot rbd-13183 in false state Feb 11 11:53:54.018: INFO: waiting for snapshot rbd-13180 (14 seconds elapsed) Feb 11 11:53:54.018: INFO: waiting for snapshot rbd-13181 (14 seconds elapsed) Feb 11 11:53:54.018: INFO: waiting for snapshot rbd-13184 (14 seconds elapsed) Feb 11 11:53:54.018: INFO: waiting for snapshot rbd-13188 (14 seconds elapsed) Feb 11 11:53:54.018: INFO: waiting for snapshot rbd-13187 (14 seconds elapsed) Feb 11 11:53:54.018: INFO: waiting for snapshot rbd-13183 (14 seconds elapsed) Feb 11 11:53:54.018: INFO: waiting for snapshot rbd-13186 (14 seconds elapsed) Feb 11 11:53:54.019: INFO: waiting for snapshot rbd-13182 (14 seconds elapsed) Feb 11 11:53:54.020: INFO: waiting for snapshot rbd-13185 (14 seconds elapsed) Feb 11 11:53:54.020: INFO: waiting for snapshot rbd-13189 (14 seconds elapsed) Feb 11 11:53:54.024: INFO: snapshot rbd-13180 in false state Feb 11 11:53:54.024: INFO: snapshot rbd-13182 in false state Feb 11 11:53:54.029: INFO: snapshot rbd-13189 in false state Feb 11 11:53:54.031: INFO: snapshot rbd-13181 in false state Feb 11 11:53:54.031: INFO: snapshot rbd-13184 in false state Feb 11 11:53:54.034: INFO: snapshot rbd-13183 in false state Feb 11 11:53:54.034: INFO: snapshot rbd-13186 in false state Feb 11 11:53:54.034: INFO: snapshot rbd-13185 in false state Feb 11 11:53:54.034: INFO: snapshot rbd-13188 in false state Feb 11 11:53:54.034: INFO: snapshot rbd-13187 in false state Feb 11 11:53:56.018: INFO: waiting for snapshot rbd-13181 (16 seconds elapsed) Feb 11 11:53:56.018: INFO: waiting for snapshot rbd-13184 (16 seconds elapsed) Feb 11 11:53:56.018: INFO: waiting for snapshot rbd-13180 (16 seconds elapsed) Feb 11 11:53:56.018: INFO: waiting for snapshot rbd-13188 (16 seconds elapsed) Feb 11 11:53:56.018: INFO: waiting for snapshot rbd-13187 (16 seconds elapsed) Feb 11 11:53:56.018: INFO: waiting for snapshot rbd-13183 (16 seconds elapsed) Feb 11 11:53:56.018: INFO: waiting for snapshot rbd-13186 (16 seconds elapsed) Feb 11 11:53:56.019: INFO: waiting for snapshot rbd-13185 (16 seconds elapsed) Feb 11 11:53:56.019: INFO: waiting for snapshot rbd-13189 (16 seconds elapsed) Feb 11 11:53:56.019: INFO: waiting for snapshot rbd-13182 (16 seconds elapsed) Feb 11 11:53:56.025: INFO: snapshot rbd-13185 in false state Feb 11 11:53:56.025: INFO: snapshot rbd-13183 in false state Feb 11 11:53:56.025: INFO: snapshot rbd-13184 in false state Feb 11 11:53:56.025: INFO: snapshot rbd-13182 in false state Feb 11 11:53:56.025: INFO: snapshot rbd-13189 in false state Feb 11 11:53:56.026: INFO: snapshot rbd-13181 in false state Feb 11 11:53:56.026: INFO: snapshot rbd-13187 in false state Feb 11 11:53:56.026: INFO: snapshot rbd-13180 in false state Feb 11 11:53:56.026: INFO: snapshot rbd-13188 in false state Feb 11 11:53:56.026: INFO: snapshot rbd-13186 in false state Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13180 (18 seconds elapsed) Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13181 (18 seconds elapsed) Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13184 (18 seconds elapsed) Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13188 (18 seconds elapsed) Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13187 (18 seconds elapsed) Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13183 (18 seconds elapsed) Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13186 (18 seconds elapsed) Feb 11 11:53:58.018: INFO: waiting for snapshot rbd-13185 (18 seconds elapsed) Feb 11 11:53:58.019: INFO: waiting for snapshot rbd-13189 (18 seconds elapsed) Feb 11 11:53:58.019: INFO: waiting for snapshot rbd-13182 (18 seconds elapsed) Feb 11 11:53:58.023: INFO: snapshot rbd-13183 in false state Feb 11 11:53:58.025: INFO: snapshot rbd-13186 in false state Feb 11 11:53:58.025: INFO: snapshot rbd-13184 in false state Feb 11 11:53:58.025: INFO: snapshot rbd-13189 in false state Feb 11 11:53:58.025: INFO: snapshot rbd-13182 in false state Feb 11 11:53:58.025: INFO: snapshot rbd-13180 in false state Feb 11 11:53:58.025: INFO: snapshot rbd-13181 in false state Feb 11 11:53:58.026: INFO: snapshot rbd-13187 in false state Feb 11 11:53:58.026: INFO: snapshot rbd-13188 in false state Feb 11 11:53:58.026: INFO: snapshot rbd-13185 in false state Feb 11 11:54:00.018: INFO: waiting for snapshot rbd-13180 (20 seconds elapsed) Feb 11 11:54:00.018: INFO: waiting for snapshot rbd-13181 (20 seconds elapsed) Feb 11 11:54:00.018: INFO: waiting for snapshot rbd-13184 (20 seconds elapsed) Feb 11 11:54:00.018: INFO: waiting for snapshot rbd-13188 (20 seconds elapsed) Feb 11 11:54:00.018: INFO: waiting for snapshot rbd-13187 (20 seconds elapsed) Feb 11 11:54:00.018: INFO: waiting for snapshot rbd-13183 (20 seconds elapsed) Feb 11 11:54:00.018: INFO: waiting for snapshot rbd-13186 (20 seconds elapsed) Feb 11 11:54:00.019: INFO: waiting for snapshot rbd-13185 (20 seconds elapsed) Feb 11 11:54:00.019: INFO: waiting for snapshot rbd-13189 (20 seconds elapsed) Feb 11 11:54:00.019: INFO: waiting for snapshot rbd-13182 (20 seconds elapsed) Feb 11 11:54:00.023: INFO: snapshot rbd-13183 in false state Feb 11 11:54:00.023: INFO: snapshot rbd-13182 in false state Feb 11 11:54:00.023: INFO: snapshot rbd-13185 in false state Feb 11 11:54:00.023: INFO: snapshot rbd-13186 in false state Feb 11 11:54:00.024: INFO: snapshot rbd-13188 in false state Feb 11 11:54:00.025: INFO: snapshot rbd-13181 in false state Feb 11 11:54:00.025: INFO: snapshot rbd-13189 in false state Feb 11 11:54:00.025: INFO: snapshot rbd-13187 in false state Feb 11 11:54:00.026: INFO: snapshot rbd-13180 in false state Feb 11 11:54:00.027: INFO: snapshot rbd-13184 in false state Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13180 (22 seconds elapsed) Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13181 (22 seconds elapsed) Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13184 (22 seconds elapsed) Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13188 (22 seconds elapsed) Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13187 (22 seconds elapsed) Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13183 (22 seconds elapsed) Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13186 (22 seconds elapsed) Feb 11 11:54:02.018: INFO: waiting for snapshot rbd-13185 (22 seconds elapsed) Feb 11 11:54:02.019: INFO: waiting for snapshot rbd-13189 (22 seconds elapsed) Feb 11 11:54:02.019: INFO: waiting for snapshot rbd-13182 (22 seconds elapsed) Feb 11 11:54:02.023: INFO: snapshot rbd-13187 in false state Feb 11 11:54:02.024: INFO: snapshot rbd-13182 in false state Feb 11 11:54:02.025: INFO: snapshot rbd-13181 in false state Feb 11 11:54:02.025: INFO: snapshot rbd-13183 in false state Feb 11 11:54:02.026: INFO: snapshot rbd-13184 in false state Feb 11 11:54:02.026: INFO: snapshot rbd-13188 in false state Feb 11 11:54:02.026: INFO: snapshot rbd-13180 in false state Feb 11 11:54:02.026: INFO: snapshot rbd-13186 in false state Feb 11 11:54:02.026: INFO: snapshot rbd-13189 in false state Feb 11 11:54:04.018: INFO: waiting for snapshot rbd-13180 (24 seconds elapsed) Feb 11 11:54:04.018: INFO: waiting for snapshot rbd-13181 (24 seconds elapsed) Feb 11 11:54:04.018: INFO: waiting for snapshot rbd-13188 (24 seconds elapsed) Feb 11 11:54:04.018: INFO: waiting for snapshot rbd-13184 (24 seconds elapsed) Feb 11 11:54:04.018: INFO: waiting for snapshot rbd-13187 (24 seconds elapsed) Feb 11 11:54:04.018: INFO: waiting for snapshot rbd-13183 (24 seconds elapsed) Feb 11 11:54:04.018: INFO: waiting for snapshot rbd-13186 (24 seconds elapsed) Feb 11 11:54:04.019: INFO: waiting for snapshot rbd-13189 (24 seconds elapsed) Feb 11 11:54:04.019: INFO: waiting for snapshot rbd-13182 (24 seconds elapsed) Feb 11 11:54:04.024: INFO: snapshot rbd-13187 in false state Feb 11 11:54:04.024: INFO: snapshot rbd-13189 in false state Feb 11 11:54:04.024: INFO: snapshot rbd-13188 in false state Feb 11 11:54:04.025: INFO: snapshot rbd-13186 in false state Feb 11 11:54:04.025: INFO: snapshot rbd-13184 in false state Feb 11 11:54:04.026: INFO: snapshot rbd-13182 in false state Feb 11 11:54:04.026: INFO: snapshot rbd-13183 in false state Feb 11 11:54:04.026: INFO: snapshot rbd-13180 in false state Feb 11 11:54:04.026: INFO: snapshot rbd-13181 in false state Feb 11 11:54:06.018: INFO: waiting for snapshot rbd-13180 (26 seconds elapsed) Feb 11 11:54:06.018: INFO: waiting for snapshot rbd-13188 (26 seconds elapsed) Feb 11 11:54:06.018: INFO: waiting for snapshot rbd-13181 (26 seconds elapsed) Feb 11 11:54:06.018: INFO: waiting for snapshot rbd-13184 (26 seconds elapsed) Feb 11 11:54:06.018: INFO: waiting for snapshot rbd-13187 (26 seconds elapsed) Feb 11 11:54:06.018: INFO: waiting for snapshot rbd-13183 (26 seconds elapsed) Feb 11 11:54:06.018: INFO: waiting for snapshot rbd-13186 (26 seconds elapsed) Feb 11 11:54:06.019: INFO: waiting for snapshot rbd-13189 (26 seconds elapsed) Feb 11 11:54:06.019: INFO: waiting for snapshot rbd-13182 (26 seconds elapsed) Feb 11 11:54:06.024: INFO: snapshot rbd-13184 in false state Feb 11 11:54:06.024: INFO: snapshot rbd-13186 in false state Feb 11 11:54:06.026: INFO: snapshot rbd-13188 in false state Feb 11 11:54:06.026: INFO: snapshot rbd-13189 in false state Feb 11 11:54:06.026: INFO: snapshot rbd-13183 in false state Feb 11 11:54:06.026: INFO: snapshot rbd-13180 in false state Feb 11 11:54:06.026: INFO: snapshot rbd-13182 in false state Feb 11 11:54:06.027: INFO: snapshot rbd-13187 in false state Feb 11 11:54:08.018: INFO: waiting for snapshot rbd-13180 (28 seconds elapsed) Feb 11 11:54:08.018: INFO: waiting for snapshot rbd-13184 (28 seconds elapsed) Feb 11 11:54:08.018: INFO: waiting for snapshot rbd-13188 (28 seconds elapsed) Feb 11 11:54:08.018: INFO: waiting for snapshot rbd-13187 (28 seconds elapsed) Feb 11 11:54:08.018: INFO: waiting for snapshot rbd-13183 (28 seconds elapsed) Feb 11 11:54:08.018: INFO: waiting for snapshot rbd-13186 (28 seconds elapsed) Feb 11 11:54:08.019: INFO: waiting for snapshot rbd-13189 (28 seconds elapsed) Feb 11 11:54:08.019: INFO: waiting for snapshot rbd-13182 (28 seconds elapsed) Feb 11 11:54:08.023: INFO: snapshot rbd-13186 in false state Feb 11 11:54:08.023: INFO: snapshot rbd-13189 in false state Feb 11 11:54:08.032: INFO: snapshot rbd-13188 in false state Feb 11 11:54:08.032: INFO: snapshot rbd-13184 in false state Feb 11 11:54:10.018: INFO: waiting for snapshot rbd-13184 (30 seconds elapsed) Feb 11 11:54:10.018: INFO: waiting for snapshot rbd-13188 (30 seconds elapsed) Feb 11 11:54:10.018: INFO: waiting for snapshot rbd-13186 (30 seconds elapsed) Feb 11 11:54:10.019: INFO: waiting for snapshot rbd-13189 (30 seconds elapsed) Feb 11 11:54:10.033: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:54:10.033: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:54:10.964: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13180 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.964: INFO: waiting for PVC rbd-13180 (0 seconds elapsed) Feb 11 11:54:10.971: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13181 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.971: INFO: waiting for PVC rbd-13181 (0 seconds elapsed) Feb 11 11:54:10.971: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13188 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.971: INFO: waiting for PVC rbd-13188 (0 seconds elapsed) Feb 11 11:54:10.971: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13185 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.971: INFO: waiting for PVC rbd-13185 (0 seconds elapsed) Feb 11 11:54:10.971: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13189 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.971: INFO: waiting for PVC rbd-13189 (0 seconds elapsed) Feb 11 11:54:10.974: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13184 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.974: INFO: waiting for PVC rbd-13184 (0 seconds elapsed) Feb 11 11:54:10.974: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13182 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.974: INFO: waiting for PVC rbd-13182 (0 seconds elapsed) Feb 11 11:54:10.974: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13183 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.975: INFO: waiting for PVC rbd-13183 (0 seconds elapsed) Feb 11 11:54:10.975: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13186 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.975: INFO: waiting for PVC rbd-13186 (0 seconds elapsed) Feb 11 11:54:10.975: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13187 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13180,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:54:10.975: INFO: waiting for PVC rbd-13187 (0 seconds elapsed) Feb 11 11:54:12.998: INFO: waiting for PVC rbd-13180 (2 seconds elapsed) Feb 11 11:54:13.003: INFO: Waiting for PV pvc-f2b16f4d-2213-4da9-966c-8839930ce20f to bind to PVC rbd-13180 Feb 11 11:54:13.004: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13180] to have phase Bound Feb 11 11:54:13.006: INFO: PersistentVolumeClaim rbd-13180 found and phase=Bound (2.489557ms) Feb 11 11:54:13.006: INFO: Waiting up to 3m0s for PersistentVolume pvc-f2b16f4d-2213-4da9-966c-8839930ce20f to have phase Bound Feb 11 11:54:13.008: INFO: PersistentVolume pvc-f2b16f4d-2213-4da9-966c-8839930ce20f found and phase=Bound (2.302309ms) Feb 11 11:54:13.013: INFO: waiting for PVC rbd-13183 (2 seconds elapsed) Feb 11 11:54:13.013: INFO: waiting for PVC rbd-13188 (2 seconds elapsed) Feb 11 11:54:13.013: INFO: waiting for PVC rbd-13189 (2 seconds elapsed) Feb 11 11:54:13.013: INFO: waiting for PVC rbd-13182 (2 seconds elapsed) Feb 11 11:54:13.013: INFO: waiting for PVC rbd-13187 (2 seconds elapsed) Feb 11 11:54:13.014: INFO: waiting for PVC rbd-13185 (2 seconds elapsed) Feb 11 11:54:13.014: INFO: waiting for PVC rbd-13186 (2 seconds elapsed) Feb 11 11:54:13.014: INFO: waiting for PVC rbd-13184 (2 seconds elapsed) Feb 11 11:54:13.021: INFO: waiting for PVC rbd-13181 (2 seconds elapsed) Feb 11 11:54:13.023: INFO: Waiting for PV pvc-e95ee0e2-6380-40bc-b414-fa126953cb0b to bind to PVC rbd-13188 Feb 11 11:54:13.023: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13188] to have phase Bound Feb 11 11:54:13.023: INFO: Waiting for PV pvc-3ff688ed-089d-4826-ad8a-0c76c4dd8a03 to bind to PVC rbd-13185 Feb 11 11:54:13.023: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13185] to have phase Bound Feb 11 11:54:13.027: INFO: Waiting up to rbd-13180 to be in Running state Feb 11 11:54:13.031: INFO: rbd-13180 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:13.034: INFO: PersistentVolumeClaim rbd-13188 found and phase=Bound (10.223395ms) Feb 11 11:54:13.034: INFO: Waiting up to 3m0s for PersistentVolume pvc-e95ee0e2-6380-40bc-b414-fa126953cb0b to have phase Bound Feb 11 11:54:13.038: INFO: PersistentVolume pvc-e95ee0e2-6380-40bc-b414-fa126953cb0b found and phase=Bound (3.896434ms) Feb 11 11:54:13.038: INFO: PersistentVolumeClaim rbd-13185 found and phase=Bound (14.138554ms) Feb 11 11:54:13.038: INFO: Waiting up to 3m0s for PersistentVolume pvc-3ff688ed-089d-4826-ad8a-0c76c4dd8a03 to have phase Bound Feb 11 11:54:13.045: INFO: PersistentVolume pvc-3ff688ed-089d-4826-ad8a-0c76c4dd8a03 found and phase=Bound (7.02132ms) Feb 11 11:54:13.057: INFO: Waiting up to rbd-13188 to be in Running state Feb 11 11:54:13.065: INFO: rbd-13188 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:13.065: INFO: Waiting up to rbd-13185 to be in Running state Feb 11 11:54:13.072: INFO: rbd-13185 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:15.013: INFO: waiting for PVC rbd-13187 (4 seconds elapsed) Feb 11 11:54:15.013: INFO: waiting for PVC rbd-13183 (4 seconds elapsed) Feb 11 11:54:15.013: INFO: waiting for PVC rbd-13182 (4 seconds elapsed) Feb 11 11:54:15.013: INFO: waiting for PVC rbd-13189 (4 seconds elapsed) Feb 11 11:54:15.014: INFO: waiting for PVC rbd-13184 (4 seconds elapsed) Feb 11 11:54:15.014: INFO: waiting for PVC rbd-13186 (4 seconds elapsed) Feb 11 11:54:15.021: INFO: waiting for PVC rbd-13181 (4 seconds elapsed) Feb 11 11:54:15.024: INFO: Waiting for PV pvc-33f5b287-120b-44d2-a121-db8fe177b43a to bind to PVC rbd-13183 Feb 11 11:54:15.024: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13183] to have phase Bound Feb 11 11:54:15.024: INFO: Waiting for PV pvc-dd6cb194-1aa0-4640-b6aa-ca462a0a6587 to bind to PVC rbd-13189 Feb 11 11:54:15.024: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13189] to have phase Bound Feb 11 11:54:15.024: INFO: Waiting for PV pvc-3b5438de-3554-47b5-88cc-a1d9f4ca2ba7 to bind to PVC rbd-13182 Feb 11 11:54:15.024: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13182] to have phase Bound Feb 11 11:54:15.026: INFO: PersistentVolumeClaim rbd-13183 found and phase=Bound (2.531623ms) Feb 11 11:54:15.026: INFO: Waiting up to 3m0s for PersistentVolume pvc-33f5b287-120b-44d2-a121-db8fe177b43a to have phase Bound Feb 11 11:54:15.027: INFO: PersistentVolumeClaim rbd-13189 found and phase=Bound (3.053672ms) Feb 11 11:54:15.027: INFO: Waiting up to 3m0s for PersistentVolume pvc-dd6cb194-1aa0-4640-b6aa-ca462a0a6587 to have phase Bound Feb 11 11:54:15.029: INFO: PersistentVolume pvc-33f5b287-120b-44d2-a121-db8fe177b43a found and phase=Bound (2.528398ms) Feb 11 11:54:15.030: INFO: PersistentVolumeClaim rbd-13182 found and phase=Bound (5.177741ms) Feb 11 11:54:15.030: INFO: Waiting up to 3m0s for PersistentVolume pvc-3b5438de-3554-47b5-88cc-a1d9f4ca2ba7 to have phase Bound Feb 11 11:54:15.031: INFO: PersistentVolume pvc-dd6cb194-1aa0-4640-b6aa-ca462a0a6587 found and phase=Bound (4.095343ms) Feb 11 11:54:15.033: INFO: PersistentVolume pvc-3b5438de-3554-47b5-88cc-a1d9f4ca2ba7 found and phase=Bound (3.356972ms) Feb 11 11:54:15.034: INFO: rbd-13180 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:15.045: INFO: Waiting up to rbd-13183 to be in Running state Feb 11 11:54:15.047: INFO: Waiting up to rbd-13189 to be in Running state Feb 11 11:54:15.047: INFO: Waiting up to rbd-13182 to be in Running state Feb 11 11:54:15.055: INFO: rbd-13183 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:15.061: INFO: rbd-13182 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:15.061: INFO: rbd-13189 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:15.078: INFO: rbd-13185 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:15.083: INFO: rbd-13188 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:17.013: INFO: waiting for PVC rbd-13187 (6 seconds elapsed) Feb 11 11:54:17.014: INFO: waiting for PVC rbd-13184 (6 seconds elapsed) Feb 11 11:54:17.014: INFO: waiting for PVC rbd-13186 (6 seconds elapsed) Feb 11 11:54:17.021: INFO: waiting for PVC rbd-13181 (6 seconds elapsed) Feb 11 11:54:17.021: INFO: Waiting for PV pvc-dc4e0c63-eaeb-4ea5-9d80-74050b428425 to bind to PVC rbd-13184 Feb 11 11:54:17.021: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13184] to have phase Bound Feb 11 11:54:17.022: INFO: Waiting for PV pvc-f6936c35-1b3a-4bd7-9a9c-285f49b7f8e3 to bind to PVC rbd-13186 Feb 11 11:54:17.022: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13186] to have phase Bound Feb 11 11:54:17.024: INFO: PersistentVolumeClaim rbd-13184 found and phase=Bound (2.907828ms) Feb 11 11:54:17.024: INFO: Waiting up to 3m0s for PersistentVolume pvc-dc4e0c63-eaeb-4ea5-9d80-74050b428425 to have phase Bound Feb 11 11:54:17.024: INFO: Waiting for PV pvc-17aee85c-3faf-491d-9ed7-f4f336e06962 to bind to PVC rbd-13187 Feb 11 11:54:17.024: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13187] to have phase Bound Feb 11 11:54:17.025: INFO: PersistentVolumeClaim rbd-13186 found and phase=Bound (2.973596ms) Feb 11 11:54:17.025: INFO: Waiting up to 3m0s for PersistentVolume pvc-f6936c35-1b3a-4bd7-9a9c-285f49b7f8e3 to have phase Bound Feb 11 11:54:17.030: INFO: PersistentVolume pvc-dc4e0c63-eaeb-4ea5-9d80-74050b428425 found and phase=Bound (6.574984ms) Feb 11 11:54:17.034: INFO: PersistentVolumeClaim rbd-13187 found and phase=Bound (9.409611ms) Feb 11 11:54:17.034: INFO: Waiting up to 3m0s for PersistentVolume pvc-17aee85c-3faf-491d-9ed7-f4f336e06962 to have phase Bound Feb 11 11:54:17.035: INFO: rbd-13180 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:17.038: INFO: Waiting for PV pvc-f275a022-5f05-4f10-a7ab-f56d4f805499 to bind to PVC rbd-13181 Feb 11 11:54:17.038: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13181] to have phase Bound Feb 11 11:54:17.040: INFO: PersistentVolume pvc-f6936c35-1b3a-4bd7-9a9c-285f49b7f8e3 found and phase=Bound (14.773286ms) Feb 11 11:54:17.042: INFO: PersistentVolume pvc-17aee85c-3faf-491d-9ed7-f4f336e06962 found and phase=Bound (8.678614ms) Feb 11 11:54:17.043: INFO: PersistentVolumeClaim rbd-13181 found and phase=Bound (5.251758ms) Feb 11 11:54:17.043: INFO: Waiting up to 3m0s for PersistentVolume pvc-f275a022-5f05-4f10-a7ab-f56d4f805499 to have phase Bound Feb 11 11:54:17.046: INFO: PersistentVolume pvc-f275a022-5f05-4f10-a7ab-f56d4f805499 found and phase=Bound (2.872238ms) Feb 11 11:54:17.050: INFO: Waiting up to rbd-13184 to be in Running state Feb 11 11:54:17.055: INFO: rbd-13184 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:17.065: INFO: Waiting up to rbd-13186 to be in Running state Feb 11 11:54:17.071: INFO: Waiting up to rbd-13187 to be in Running state Feb 11 11:54:17.071: INFO: Waiting up to rbd-13181 to be in Running state Feb 11 11:54:17.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:17.075: INFO: rbd-13188 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:17.076: INFO: rbd-13182 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:17.076: INFO: rbd-13186 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:17.076: INFO: rbd-13189 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:17.084: INFO: rbd-13181 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:17.090: INFO: rbd-13187 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:54:17.091: INFO: rbd-13185 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:19.035: INFO: rbd-13180 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:19.058: INFO: rbd-13183 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:19.058: INFO: rbd-13184 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:19.063: INFO: rbd-13182 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:19.066: INFO: rbd-13189 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:19.067: INFO: rbd-13188 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:19.075: INFO: rbd-13185 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:19.078: INFO: rbd-13186 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:19.087: INFO: rbd-13181 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:19.093: INFO: rbd-13187 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:54:21.035: INFO: rbd-13180 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:21.058: INFO: rbd-13183 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:21.058: INFO: rbd-13184 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:21.064: INFO: rbd-13182 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:21.065: INFO: rbd-13189 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:21.067: INFO: rbd-13188 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:21.075: INFO: rbd-13185 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:21.078: INFO: rbd-13186 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:21.087: INFO: rbd-13181 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:21.093: INFO: rbd-13187 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:54:23.035: INFO: rbd-13180 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:23.061: INFO: rbd-13183 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:23.061: INFO: rbd-13184 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:23.064: INFO: rbd-13182 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:23.065: INFO: rbd-13189 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:23.076: INFO: rbd-13185 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:23.081: INFO: rbd-13186 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:23.088: INFO: rbd-13181 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:23.094: INFO: rbd-13187 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:54:25.036: INFO: rbd-13180 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:25.059: INFO: rbd-13183 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:25.059: INFO: rbd-13184 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:25.064: INFO: rbd-13189 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:25.065: INFO: rbd-13182 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:25.076: INFO: rbd-13185 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:25.079: INFO: rbd-13186 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:25.088: INFO: rbd-13181 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:25.094: INFO: rbd-13187 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:54:27.035: INFO: rbd-13180 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:54:27.058: INFO: rbd-13184 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:27.058: INFO: rbd-13183 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:27.064: INFO: rbd-13182 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:27.079: INFO: rbd-13186 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:27.087: INFO: rbd-13181 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:27.093: INFO: rbd-13187 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:54:29.058: INFO: rbd-13184 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:29.058: INFO: rbd-13183 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:54:29.063: INFO: rbd-13182 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:54:29.079: INFO: rbd-13186 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:29.087: INFO: rbd-13181 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:29.094: INFO: rbd-13187 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:54:31.058: INFO: rbd-13184 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:54:31.058: INFO: rbd-13183 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:54:31.064: INFO: rbd-13182 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:54:31.078: INFO: rbd-13186 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:54:31.087: INFO: rbd-13181 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:54:33.058: INFO: rbd-13183 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:54:33.058: INFO: rbd-13184 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:54:33.080: INFO: rbd-13186 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:54:33.087: INFO: rbd-13181 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:54:35.059: INFO: rbd-13184 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:54:35.080: INFO: rbd-13186 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:54:35.088: INFO: rbd-13181 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:54:37.062: INFO: rbd-13184 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:54:37.079: INFO: rbd-13186 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:54:39.058: INFO: rbd-13184 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:54:41.065: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:54:41.065: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:54:41.989: INFO: Waiting for pod rbd-13188 to be deleted Feb 11 11:54:41.990: INFO: Waiting for pod rbd-13186 to be deleted Feb 11 11:54:41.990: INFO: Waiting for pod rbd-13187 to be deleted Feb 11 11:54:41.993: INFO: Waiting for pod rbd-13183 to be deleted Feb 11 11:54:42.008: INFO: Waiting for pod rbd-13182 to be deleted Feb 11 11:54:42.008: INFO: rbd-13186 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.008: INFO: Waiting for pod rbd-13185 to be deleted Feb 11 11:54:42.009: INFO: Waiting for pod rbd-13189 to be deleted Feb 11 11:54:42.009: INFO: Waiting for pod rbd-13184 to be deleted Feb 11 11:54:42.009: INFO: Waiting for pod rbd-13180 to be deleted Feb 11 11:54:42.009: INFO: Waiting for pod rbd-13181 to be deleted Feb 11 11:54:42.017: INFO: rbd-13188 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.017: INFO: rbd-13187 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.017: INFO: rbd-13183 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.022: INFO: rbd-13184 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.022: INFO: rbd-13189 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.024: INFO: rbd-13181 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.024: INFO: rbd-13185 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.024: INFO: rbd-13182 app to be deleted (0 seconds elapsed) Feb 11 11:54:42.024: INFO: rbd-13180 app to be deleted (0 seconds elapsed) Feb 11 11:54:44.011: INFO: rbd-13186 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.021: INFO: rbd-13187 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.021: INFO: rbd-13183 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.021: INFO: rbd-13188 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.026: INFO: rbd-13189 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.027: INFO: rbd-13184 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.027: INFO: rbd-13185 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.027: INFO: rbd-13182 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.028: INFO: rbd-13181 app to be deleted (2 seconds elapsed) Feb 11 11:54:44.029: INFO: rbd-13180 app to be deleted (2 seconds elapsed) Feb 11 11:54:46.012: INFO: rbd-13186 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.021: INFO: rbd-13188 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.021: INFO: rbd-13183 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.021: INFO: rbd-13187 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.027: INFO: rbd-13189 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.027: INFO: rbd-13181 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.027: INFO: Deleting PersistentVolumeClaim rbd-13182 on namespace rbd-1318 Feb 11 11:54:46.027: INFO: rbd-13184 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.028: INFO: rbd-13185 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.028: INFO: rbd-13180 app to be deleted (4 seconds elapsed) Feb 11 11:54:46.037: INFO: waiting for PVC rbd-13182 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:48.013: INFO: rbd-13186 app to be deleted (6 seconds elapsed) Feb 11 11:54:48.021: INFO: Deleting PersistentVolumeClaim rbd-13183 on namespace rbd-1318 Feb 11 11:54:48.021: INFO: rbd-13187 app to be deleted (6 seconds elapsed) Feb 11 11:54:48.022: INFO: rbd-13188 app to be deleted (6 seconds elapsed) Feb 11 11:54:48.026: INFO: Deleting PersistentVolumeClaim rbd-13189 on namespace rbd-1318 Feb 11 11:54:48.026: INFO: rbd-13184 app to be deleted (6 seconds elapsed) Feb 11 11:54:48.029: INFO: Deleting PersistentVolumeClaim rbd-13185 on namespace rbd-1318 Feb 11 11:54:48.030: INFO: rbd-13181 app to be deleted (6 seconds elapsed) Feb 11 11:54:48.030: INFO: rbd-13180 app to be deleted (6 seconds elapsed) Feb 11 11:54:48.046: INFO: waiting for PVC rbd-13182 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:48.055: INFO: waiting for PVC rbd-13185 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:48.056: INFO: waiting for PVC rbd-13189 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:48.056: INFO: waiting for PVC rbd-13183 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:50.014: INFO: Deleting PersistentVolumeClaim rbd-13186 on namespace rbd-1318 Feb 11 11:54:50.021: INFO: Deleting PersistentVolumeClaim rbd-13188 on namespace rbd-1318 Feb 11 11:54:50.022: INFO: rbd-13187 app to be deleted (8 seconds elapsed) Feb 11 11:54:50.028: INFO: Deleting PersistentVolumeClaim rbd-13180 on namespace rbd-1318 Feb 11 11:54:50.029: INFO: rbd-13181 app to be deleted (8 seconds elapsed) Feb 11 11:54:50.029: INFO: Deleting PersistentVolumeClaim rbd-13184 on namespace rbd-1318 Feb 11 11:54:50.030: INFO: waiting for PVC rbd-13186 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:50.067: INFO: waiting for PVC rbd-13185 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:50.068: INFO: waiting for PVC rbd-13189 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:50.069: INFO: waiting for PVC rbd-13183 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:50.069: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:50.074: INFO: waiting for PVC rbd-13180 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:50.085: INFO: waiting for PVC rbd-13184 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:54:52.027: INFO: rbd-13181 app to be deleted (10 seconds elapsed) Feb 11 11:54:52.027: INFO: rbd-13187 app to be deleted (10 seconds elapsed) Feb 11 11:54:52.052: INFO: waiting for PVC rbd-13186 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:52.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:52.090: INFO: waiting for PVC rbd-13180 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:52.094: INFO: waiting for PVC rbd-13184 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:54:54.021: INFO: rbd-13187 app to be deleted (12 seconds elapsed) Feb 11 11:54:54.027: INFO: rbd-13181 app to be deleted (12 seconds elapsed) Feb 11 11:54:54.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (4 seconds elapsed) Feb 11 11:54:56.021: INFO: rbd-13187 app to be deleted (14 seconds elapsed) Feb 11 11:54:56.027: INFO: rbd-13181 app to be deleted (14 seconds elapsed) Feb 11 11:54:56.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (6 seconds elapsed) Feb 11 11:54:58.021: INFO: rbd-13187 app to be deleted (16 seconds elapsed) Feb 11 11:54:58.026: INFO: rbd-13181 app to be deleted (16 seconds elapsed) Feb 11 11:54:58.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (8 seconds elapsed) Feb 11 11:55:00.022: INFO: rbd-13187 app to be deleted (18 seconds elapsed) Feb 11 11:55:00.027: INFO: Deleting PersistentVolumeClaim rbd-13181 on namespace rbd-1318 Feb 11 11:55:00.039: INFO: waiting for PVC rbd-13181 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:55:00.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (10 seconds elapsed) Feb 11 11:55:02.021: INFO: rbd-13187 app to be deleted (20 seconds elapsed) Feb 11 11:55:02.043: INFO: waiting for PVC rbd-13181 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:55:02.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (12 seconds elapsed) Feb 11 11:55:04.020: INFO: rbd-13187 app to be deleted (22 seconds elapsed) Feb 11 11:55:04.088: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (14 seconds elapsed) Feb 11 11:55:06.021: INFO: rbd-13187 app to be deleted (24 seconds elapsed) Feb 11 11:55:06.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (16 seconds elapsed) Feb 11 11:55:08.021: INFO: rbd-13187 app to be deleted (26 seconds elapsed) Feb 11 11:55:08.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (18 seconds elapsed) Feb 11 11:55:10.021: INFO: Deleting PersistentVolumeClaim rbd-13187 on namespace rbd-1318 Feb 11 11:55:10.035: INFO: waiting for PVC rbd-13187 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:55:10.089: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (20 seconds elapsed) Feb 11 11:55:12.041: INFO: waiting for PVC rbd-13187 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:55:12.053: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:55:12.053: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:12.967: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13189 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.967: INFO: waiting for PVC rbd-13189 (0 seconds elapsed) Feb 11 11:55:12.972: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13184 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.972: INFO: waiting for PVC rbd-13184 (0 seconds elapsed) Feb 11 11:55:12.973: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13180 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.973: INFO: waiting for PVC rbd-13180 (0 seconds elapsed) Feb 11 11:55:12.973: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13183 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.973: INFO: waiting for PVC rbd-13183 (0 seconds elapsed) Feb 11 11:55:12.982: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13185 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.983: INFO: waiting for PVC rbd-13185 (0 seconds elapsed) Feb 11 11:55:12.984: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13188 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.984: INFO: waiting for PVC rbd-13188 (0 seconds elapsed) Feb 11 11:55:12.985: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13186 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.985: INFO: waiting for PVC rbd-13186 (0 seconds elapsed) Feb 11 11:55:12.990: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13181 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.990: INFO: waiting for PVC rbd-13181 (0 seconds elapsed) Feb 11 11:55:12.990: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13182 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.990: INFO: waiting for PVC rbd-13182 (0 seconds elapsed) Feb 11 11:55:12.990: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-13187 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-13188,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Feb 11 11:55:12.990: INFO: waiting for PVC rbd-13187 (0 seconds elapsed) Feb 11 11:55:14.983: INFO: waiting for PVC rbd-13189 (2 seconds elapsed) Feb 11 11:55:15.007: INFO: waiting for PVC rbd-13180 (2 seconds elapsed) Feb 11 11:55:15.007: INFO: waiting for PVC rbd-13183 (2 seconds elapsed) Feb 11 11:55:15.007: INFO: waiting for PVC rbd-13184 (2 seconds elapsed) Feb 11 11:55:15.024: INFO: Waiting for PV pvc-2961cc44-d78f-4dd5-aeba-2873ccef5862 to bind to PVC rbd-13184 Feb 11 11:55:15.024: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13184] to have phase Bound Feb 11 11:55:15.028: INFO: Waiting for PV pvc-6e8d3e31-c4af-4f12-8380-0d60efbbb8a4 to bind to PVC rbd-13183 Feb 11 11:55:15.028: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13183] to have phase Bound Feb 11 11:55:15.029: INFO: PersistentVolumeClaim rbd-13184 found and phase=Bound (5.164075ms) Feb 11 11:55:15.029: INFO: Waiting up to 3m0s for PersistentVolume pvc-2961cc44-d78f-4dd5-aeba-2873ccef5862 to have phase Bound Feb 11 11:55:15.037: INFO: waiting for PVC rbd-13185 (2 seconds elapsed) Feb 11 11:55:15.038: INFO: waiting for PVC rbd-13181 (2 seconds elapsed) Feb 11 11:55:15.038: INFO: waiting for PVC rbd-13186 (2 seconds elapsed) Feb 11 11:55:15.038: INFO: waiting for PVC rbd-13187 (2 seconds elapsed) Feb 11 11:55:15.040: INFO: PersistentVolumeClaim rbd-13183 found and phase=Bound (12.145706ms) Feb 11 11:55:15.040: INFO: Waiting up to 3m0s for PersistentVolume pvc-6e8d3e31-c4af-4f12-8380-0d60efbbb8a4 to have phase Bound Feb 11 11:55:15.041: INFO: PersistentVolume pvc-2961cc44-d78f-4dd5-aeba-2873ccef5862 found and phase=Bound (11.277074ms) Feb 11 11:55:15.041: INFO: waiting for PVC rbd-13182 (2 seconds elapsed) Feb 11 11:55:15.045: INFO: waiting for PVC rbd-13188 (2 seconds elapsed) Feb 11 11:55:15.046: INFO: PersistentVolume pvc-6e8d3e31-c4af-4f12-8380-0d60efbbb8a4 found and phase=Bound (6.032842ms) Feb 11 11:55:15.060: INFO: Waiting up to rbd-13184 to be in Running state Feb 11 11:55:15.063: INFO: Waiting up to rbd-13183 to be in Running state Feb 11 11:55:15.068: INFO: rbd-13183 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:15.068: INFO: rbd-13184 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:16.983: INFO: waiting for PVC rbd-13189 (4 seconds elapsed) Feb 11 11:55:16.989: INFO: Waiting for PV pvc-3df8f43a-d9b4-4dc0-9541-7103000b2713 to bind to PVC rbd-13189 Feb 11 11:55:16.989: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13189] to have phase Bound Feb 11 11:55:16.992: INFO: PersistentVolumeClaim rbd-13189 found and phase=Bound (2.576886ms) Feb 11 11:55:16.992: INFO: Waiting up to 3m0s for PersistentVolume pvc-3df8f43a-d9b4-4dc0-9541-7103000b2713 to have phase Bound Feb 11 11:55:16.995: INFO: PersistentVolume pvc-3df8f43a-d9b4-4dc0-9541-7103000b2713 found and phase=Bound (2.7867ms) Feb 11 11:55:17.006: INFO: Waiting up to rbd-13189 to be in Running state Feb 11 11:55:17.008: INFO: waiting for PVC rbd-13180 (4 seconds elapsed) Feb 11 11:55:17.009: INFO: rbd-13189 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:17.017: INFO: Waiting for PV pvc-5eafb984-86c3-4490-9122-2cfd91b9daa3 to bind to PVC rbd-13180 Feb 11 11:55:17.017: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13180] to have phase Bound Feb 11 11:55:17.022: INFO: PersistentVolumeClaim rbd-13180 found and phase=Bound (5.234725ms) Feb 11 11:55:17.022: INFO: Waiting up to 3m0s for PersistentVolume pvc-5eafb984-86c3-4490-9122-2cfd91b9daa3 to have phase Bound Feb 11 11:55:17.029: INFO: PersistentVolume pvc-5eafb984-86c3-4490-9122-2cfd91b9daa3 found and phase=Bound (7.422574ms) Feb 11 11:55:17.036: INFO: waiting for PVC rbd-13185 (4 seconds elapsed) Feb 11 11:55:17.037: INFO: waiting for PVC rbd-13186 (4 seconds elapsed) Feb 11 11:55:17.037: INFO: waiting for PVC rbd-13187 (4 seconds elapsed) Feb 11 11:55:17.037: INFO: waiting for PVC rbd-13181 (4 seconds elapsed) Feb 11 11:55:17.041: INFO: waiting for PVC rbd-13182 (4 seconds elapsed) Feb 11 11:55:17.045: INFO: waiting for PVC rbd-13188 (4 seconds elapsed) Feb 11 11:55:17.059: INFO: Waiting up to rbd-13180 to be in Running state Feb 11 11:55:17.065: INFO: rbd-13180 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:17.066: INFO: Waiting for PV pvc-e93be23c-1815-4610-87b7-4fa58634284e to bind to PVC rbd-13182 Feb 11 11:55:17.066: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13182] to have phase Bound Feb 11 11:55:17.067: INFO: Waiting for PV pvc-9b1b5631-24ee-48a5-b125-4be5144f7e9d to bind to PVC rbd-13185 Feb 11 11:55:17.067: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13185] to have phase Bound Feb 11 11:55:17.068: INFO: Waiting for PV pvc-a9a6585c-8792-4918-a806-7f94f4ff66e1 to bind to PVC rbd-13188 Feb 11 11:55:17.068: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13188] to have phase Bound Feb 11 11:55:17.072: INFO: PersistentVolumeClaim rbd-13185 found and phase=Bound (4.900349ms) Feb 11 11:55:17.072: INFO: Waiting up to 3m0s for PersistentVolume pvc-9b1b5631-24ee-48a5-b125-4be5144f7e9d to have phase Bound Feb 11 11:55:17.072: INFO: PersistentVolumeClaim rbd-13182 found and phase=Bound (5.573538ms) Feb 11 11:55:17.072: INFO: Waiting up to 3m0s for PersistentVolume pvc-e93be23c-1815-4610-87b7-4fa58634284e to have phase Bound Feb 11 11:55:17.073: INFO: rbd-13183 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:17.073: INFO: PersistentVolumeClaim rbd-13188 found and phase=Bound (5.585443ms) Feb 11 11:55:17.073: INFO: Waiting up to 3m0s for PersistentVolume pvc-a9a6585c-8792-4918-a806-7f94f4ff66e1 to have phase Bound Feb 11 11:55:17.073: INFO: rbd-13184 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:17.080: INFO: PersistentVolume pvc-e93be23c-1815-4610-87b7-4fa58634284e found and phase=Bound (8.503869ms) Feb 11 11:55:17.081: INFO: PersistentVolume pvc-a9a6585c-8792-4918-a806-7f94f4ff66e1 found and phase=Bound (7.379344ms) Feb 11 11:55:17.081: INFO: PersistentVolume pvc-9b1b5631-24ee-48a5-b125-4be5144f7e9d found and phase=Bound (8.810054ms) Feb 11 11:55:17.099: INFO: Waiting up to rbd-13185 to be in Running state Feb 11 11:55:17.102: INFO: Waiting up to rbd-13188 to be in Running state Feb 11 11:55:17.102: INFO: rbd-13185 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:17.102: INFO: Waiting up to rbd-13182 to be in Running state Feb 11 11:55:17.108: INFO: rbd-13188 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:17.114: INFO: rbd-13182 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:19.013: INFO: rbd-13189 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:19.037: INFO: waiting for PVC rbd-13186 (6 seconds elapsed) Feb 11 11:55:19.037: INFO: waiting for PVC rbd-13181 (6 seconds elapsed) Feb 11 11:55:19.037: INFO: waiting for PVC rbd-13187 (6 seconds elapsed) Feb 11 11:55:19.044: INFO: Waiting for PV pvc-d3ca6ef4-da69-46b6-8462-60d1442f5cd8 to bind to PVC rbd-13181 Feb 11 11:55:19.044: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13181] to have phase Bound Feb 11 11:55:19.044: INFO: Waiting for PV pvc-79272cc0-5a67-4854-9b42-ee40a315b872 to bind to PVC rbd-13186 Feb 11 11:55:19.044: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13186] to have phase Bound Feb 11 11:55:19.045: INFO: Waiting for PV pvc-2ddb662e-3d1d-4179-9a21-54a6261ff484 to bind to PVC rbd-13187 Feb 11 11:55:19.045: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-13187] to have phase Bound Feb 11 11:55:19.047: INFO: PersistentVolumeClaim rbd-13181 found and phase=Bound (3.420697ms) Feb 11 11:55:19.047: INFO: Waiting up to 3m0s for PersistentVolume pvc-d3ca6ef4-da69-46b6-8462-60d1442f5cd8 to have phase Bound Feb 11 11:55:19.047: INFO: PersistentVolumeClaim rbd-13186 found and phase=Bound (3.434423ms) Feb 11 11:55:19.047: INFO: Waiting up to 3m0s for PersistentVolume pvc-79272cc0-5a67-4854-9b42-ee40a315b872 to have phase Bound Feb 11 11:55:19.048: INFO: PersistentVolumeClaim rbd-13187 found and phase=Bound (2.762233ms) Feb 11 11:55:19.048: INFO: Waiting up to 3m0s for PersistentVolume pvc-2ddb662e-3d1d-4179-9a21-54a6261ff484 to have phase Bound Feb 11 11:55:19.049: INFO: PersistentVolume pvc-d3ca6ef4-da69-46b6-8462-60d1442f5cd8 found and phase=Bound (2.154158ms) Feb 11 11:55:19.050: INFO: PersistentVolume pvc-2ddb662e-3d1d-4179-9a21-54a6261ff484 found and phase=Bound (2.456056ms) Feb 11 11:55:19.051: INFO: PersistentVolume pvc-79272cc0-5a67-4854-9b42-ee40a315b872 found and phase=Bound (3.392725ms) Feb 11 11:55:19.060: INFO: Waiting up to rbd-13181 to be in Running state Feb 11 11:55:19.063: INFO: rbd-13181 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:19.066: INFO: Waiting up to rbd-13186 to be in Running state Feb 11 11:55:19.074: INFO: rbd-13186 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:19.074: INFO: Waiting up to rbd-13187 to be in Running state Feb 11 11:55:19.074: INFO: rbd-13180 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:19.080: INFO: rbd-13187 app is in Pending phase expected to be in Running state (0 seconds elapsed) Feb 11 11:55:19.080: INFO: rbd-13184 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:19.080: INFO: rbd-13183 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:19.114: INFO: rbd-13185 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:19.117: INFO: rbd-13188 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:19.118: INFO: rbd-13182 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:21.014: INFO: rbd-13189 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:21.066: INFO: rbd-13181 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:21.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:21.071: INFO: rbd-13184 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:21.072: INFO: rbd-13183 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:21.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:21.083: INFO: rbd-13187 app is in Pending phase expected to be in Running state (2 seconds elapsed) Feb 11 11:55:21.105: INFO: rbd-13185 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:21.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:21.117: INFO: rbd-13182 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:23.013: INFO: rbd-13189 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:23.067: INFO: rbd-13181 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:23.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:23.071: INFO: rbd-13184 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:23.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:23.076: INFO: rbd-13186 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:23.082: INFO: rbd-13187 app is in Pending phase expected to be in Running state (4 seconds elapsed) Feb 11 11:55:23.105: INFO: rbd-13185 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:23.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:23.116: INFO: rbd-13182 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:25.013: INFO: rbd-13189 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:25.068: INFO: rbd-13181 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:25.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:25.071: INFO: rbd-13184 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:25.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:25.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:25.082: INFO: rbd-13187 app is in Pending phase expected to be in Running state (6 seconds elapsed) Feb 11 11:55:25.105: INFO: rbd-13185 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:25.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:25.116: INFO: rbd-13182 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:27.012: INFO: rbd-13189 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:27.066: INFO: rbd-13181 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:27.067: INFO: rbd-13180 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:27.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:27.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:27.076: INFO: rbd-13186 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:27.085: INFO: rbd-13187 app is in Pending phase expected to be in Running state (8 seconds elapsed) Feb 11 11:55:27.105: INFO: rbd-13185 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:27.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:27.117: INFO: rbd-13182 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:29.012: INFO: rbd-13189 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:29.067: INFO: rbd-13181 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:29.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:29.072: INFO: rbd-13183 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:29.074: INFO: rbd-13184 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:29.076: INFO: rbd-13186 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:29.082: INFO: rbd-13187 app is in Pending phase expected to be in Running state (10 seconds elapsed) Feb 11 11:55:29.105: INFO: rbd-13185 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:29.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:29.116: INFO: rbd-13182 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:31.012: INFO: rbd-13189 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:31.066: INFO: rbd-13181 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:31.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:31.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:31.071: INFO: rbd-13184 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:31.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:31.082: INFO: rbd-13187 app is in Pending phase expected to be in Running state (12 seconds elapsed) Feb 11 11:55:31.104: INFO: rbd-13185 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:31.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:31.116: INFO: rbd-13182 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:33.013: INFO: rbd-13189 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:33.066: INFO: rbd-13181 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:33.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:33.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:33.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:33.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:33.082: INFO: rbd-13187 app is in Pending phase expected to be in Running state (14 seconds elapsed) Feb 11 11:55:33.105: INFO: rbd-13185 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:33.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:33.117: INFO: rbd-13182 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:35.012: INFO: rbd-13189 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:35.068: INFO: rbd-13181 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:35.070: INFO: rbd-13180 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:35.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:35.074: INFO: rbd-13183 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:35.078: INFO: rbd-13186 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:35.083: INFO: rbd-13187 app is in Pending phase expected to be in Running state (16 seconds elapsed) Feb 11 11:55:35.108: INFO: rbd-13185 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:35.114: INFO: rbd-13188 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:35.117: INFO: rbd-13182 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:37.013: INFO: rbd-13189 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:37.067: INFO: rbd-13181 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:37.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:37.072: INFO: rbd-13183 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:37.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:37.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:37.084: INFO: rbd-13187 app is in Pending phase expected to be in Running state (18 seconds elapsed) Feb 11 11:55:37.106: INFO: rbd-13185 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:37.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:37.117: INFO: rbd-13182 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:39.067: INFO: rbd-13181 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:39.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:39.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:39.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:39.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:39.085: INFO: rbd-13187 app is in Pending phase expected to be in Running state (20 seconds elapsed) Feb 11 11:55:39.105: INFO: rbd-13185 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:39.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:39.116: INFO: rbd-13182 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:41.067: INFO: rbd-13181 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:41.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:41.071: INFO: rbd-13183 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:41.071: INFO: rbd-13184 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:41.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (22 seconds elapsed) Feb 11 11:55:41.104: INFO: rbd-13185 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:41.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:41.116: INFO: rbd-13182 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:43.067: INFO: rbd-13181 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:43.068: INFO: rbd-13180 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:43.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (28 seconds elapsed) Feb 11 11:55:43.076: INFO: rbd-13186 app is in Pending phase expected to be in Running state (24 seconds elapsed) Feb 11 11:55:43.115: INFO: rbd-13185 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:43.117: INFO: rbd-13182 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:43.117: INFO: rbd-13188 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:45.068: INFO: rbd-13181 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:45.071: INFO: rbd-13184 app is in Pending phase expected to be in Running state (30 seconds elapsed) Feb 11 11:55:45.079: INFO: rbd-13186 app is in Pending phase expected to be in Running state (26 seconds elapsed) Feb 11 11:55:45.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (28 seconds elapsed) Feb 11 11:55:45.116: INFO: rbd-13182 app is in Pending phase expected to be in Running state (28 seconds elapsed) Feb 11 11:55:47.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (32 seconds elapsed) Feb 11 11:55:47.077: INFO: rbd-13186 app is in Pending phase expected to be in Running state (28 seconds elapsed) Feb 11 11:55:47.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (30 seconds elapsed) Feb 11 11:55:49.071: INFO: rbd-13184 app is in Pending phase expected to be in Running state (34 seconds elapsed) Feb 11 11:55:49.111: INFO: rbd-13188 app is in Pending phase expected to be in Running state (32 seconds elapsed) Feb 11 11:55:51.072: INFO: rbd-13184 app is in Pending phase expected to be in Running state (36 seconds elapsed) Feb 11 11:55:53.078: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:55:53.078: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:53.969: INFO: Deleting PersistentVolumeClaim rbd-pvc on namespace rbd-1318 Feb 11 11:55:53.982: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:55:55.986: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:55:55.999: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:55:55.999: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.912: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.912: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.912: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.912: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.912: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.912: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.914: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.914: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.914: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.914: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:55:56.925: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13181 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.925: INFO: deleting snapshot rbd-13181 (0 seconds elapsed) Feb 11 11:55:56.935: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13189 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.935: INFO: deleting snapshot rbd-13189 (0 seconds elapsed) Feb 11 11:55:56.936: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13182 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.936: INFO: deleting snapshot rbd-13182 (0 seconds elapsed) Feb 11 11:55:56.936: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13180 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.936: INFO: deleting snapshot rbd-13180 (0 seconds elapsed) Feb 11 11:55:56.936: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13187 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.936: INFO: deleting snapshot rbd-13187 (0 seconds elapsed) Feb 11 11:55:56.936: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13185 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.936: INFO: deleting snapshot rbd-13185 (0 seconds elapsed) Feb 11 11:55:56.936: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13188 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.936: INFO: deleting snapshot rbd-13188 (0 seconds elapsed) Feb 11 11:55:56.936: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13183 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.936: INFO: deleting snapshot rbd-13183 (0 seconds elapsed) Feb 11 11:55:56.947: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13186 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.947: INFO: deleting snapshot rbd-13186 (0 seconds elapsed) Feb 11 11:55:56.947: INFO: Waiting up to &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-13184 rbd-1318 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc0002d3c20 } 0xc00075cd30} } to be deleted Feb 11 11:55:56.947: INFO: deleting snapshot rbd-13184 (0 seconds elapsed) Feb 11 11:55:58.946: INFO: deleting snapshot rbd-13181 (2 seconds elapsed) Feb 11 11:55:58.958: INFO: deleting snapshot rbd-13187 (2 seconds elapsed) Feb 11 11:55:58.958: INFO: deleting snapshot rbd-13188 (2 seconds elapsed) Feb 11 11:55:58.958: INFO: deleting snapshot rbd-13185 (2 seconds elapsed) Feb 11 11:55:58.958: INFO: deleting snapshot rbd-13180 (2 seconds elapsed) Feb 11 11:55:58.959: INFO: deleting snapshot rbd-13182 (2 seconds elapsed) Feb 11 11:55:58.961: INFO: deleting snapshot rbd-13189 (2 seconds elapsed) Feb 11 11:55:58.962: INFO: deleting snapshot rbd-13183 (2 seconds elapsed) Feb 11 11:55:58.965: INFO: deleting snapshot rbd-13186 (2 seconds elapsed) Feb 11 11:55:58.971: INFO: deleting snapshot rbd-13184 (2 seconds elapsed) Feb 11 11:56:00.947: INFO: deleting snapshot rbd-13181 (4 seconds elapsed) Feb 11 11:56:00.958: INFO: deleting snapshot rbd-13188 (4 seconds elapsed) Feb 11 11:56:00.958: INFO: deleting snapshot rbd-13182 (4 seconds elapsed) Feb 11 11:56:00.958: INFO: deleting snapshot rbd-13187 (4 seconds elapsed) Feb 11 11:56:00.958: INFO: deleting snapshot rbd-13180 (4 seconds elapsed) Feb 11 11:56:00.958: INFO: deleting snapshot rbd-13185 (4 seconds elapsed) Feb 11 11:56:00.962: INFO: deleting snapshot rbd-13189 (4 seconds elapsed) Feb 11 11:56:00.962: INFO: deleting snapshot rbd-13183 (4 seconds elapsed) Feb 11 11:56:00.965: INFO: deleting snapshot rbd-13186 (4 seconds elapsed) Feb 11 11:56:00.971: INFO: deleting snapshot rbd-13184 (4 seconds elapsed) Feb 11 11:56:02.947: INFO: deleting snapshot rbd-13181 (6 seconds elapsed) Feb 11 11:56:02.958: INFO: deleting snapshot rbd-13180 (6 seconds elapsed) Feb 11 11:56:02.958: INFO: deleting snapshot rbd-13188 (6 seconds elapsed) Feb 11 11:56:02.958: INFO: deleting snapshot rbd-13187 (6 seconds elapsed) Feb 11 11:56:02.958: INFO: deleting snapshot rbd-13182 (6 seconds elapsed) Feb 11 11:56:02.958: INFO: deleting snapshot rbd-13185 (6 seconds elapsed) Feb 11 11:56:02.961: INFO: deleting snapshot rbd-13183 (6 seconds elapsed) Feb 11 11:56:02.961: INFO: deleting snapshot rbd-13189 (6 seconds elapsed) Feb 11 11:56:02.965: INFO: deleting snapshot rbd-13186 (6 seconds elapsed) Feb 11 11:56:02.971: INFO: deleting snapshot rbd-13184 (6 seconds elapsed) Feb 11 11:56:04.958: INFO: deleting snapshot rbd-13188 (8 seconds elapsed) Feb 11 11:56:04.958: INFO: deleting snapshot rbd-13182 (8 seconds elapsed) Feb 11 11:56:04.958: INFO: deleting snapshot rbd-13187 (8 seconds elapsed) Feb 11 11:56:04.958: INFO: deleting snapshot rbd-13180 (8 seconds elapsed) Feb 11 11:56:04.961: INFO: deleting snapshot rbd-13189 (8 seconds elapsed) Feb 11 11:56:04.965: INFO: deleting snapshot rbd-13186 (8 seconds elapsed) Feb 11 11:56:04.971: INFO: deleting snapshot rbd-13184 (8 seconds elapsed) Feb 11 11:56:06.958: INFO: deleting snapshot rbd-13188 (10 seconds elapsed) Feb 11 11:56:06.961: INFO: deleting snapshot rbd-13189 (10 seconds elapsed) Feb 11 11:56:06.965: INFO: deleting snapshot rbd-13186 (10 seconds elapsed) Feb 11 11:56:06.971: INFO: deleting snapshot rbd-13184 (10 seconds elapsed) Feb 11 11:56:06.980: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-xqj45 ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Feb 11 11:56:06.980: INFO: >>> kubeConfig: /root/.kube/config Feb 11 11:56:07.891: INFO: Waiting for pod rbd-13188 to be deleted Feb 11 11:56:07.895: INFO: Waiting for pod rbd-13187 to be deleted Feb 11 11:56:07.896: INFO: Waiting for pod rbd-13186 to be deleted Feb 11 11:56:07.896: INFO: Waiting for pod rbd-13182 to be deleted Feb 11 11:56:07.903: INFO: Waiting for pod rbd-13189 to be deleted Feb 11 11:56:07.903: INFO: Waiting for pod rbd-13183 to be deleted Feb 11 11:56:07.903: INFO: Waiting for pod rbd-13184 to be deleted Feb 11 11:56:07.914: INFO: rbd-13182 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.914: INFO: rbd-13188 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.922: INFO: rbd-13187 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.924: INFO: rbd-13186 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.934: INFO: Waiting for pod rbd-13185 to be deleted Feb 11 11:56:07.934: INFO: Waiting for pod rbd-13180 to be deleted Feb 11 11:56:07.937: INFO: rbd-13184 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.951: INFO: Waiting for pod rbd-13181 to be deleted Feb 11 11:56:07.952: INFO: rbd-13189 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.967: INFO: rbd-13183 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.967: INFO: rbd-13185 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.968: INFO: rbd-13180 app to be deleted (0 seconds elapsed) Feb 11 11:56:07.975: INFO: rbd-13181 app to be deleted (0 seconds elapsed) Feb 11 11:56:09.918: INFO: rbd-13188 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.919: INFO: rbd-13182 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.925: INFO: rbd-13187 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.927: INFO: rbd-13186 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.941: INFO: rbd-13184 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.955: INFO: rbd-13189 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.970: INFO: rbd-13185 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.971: INFO: rbd-13180 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.971: INFO: rbd-13183 app to be deleted (2 seconds elapsed) Feb 11 11:56:09.978: INFO: rbd-13181 app to be deleted (2 seconds elapsed) Feb 11 11:56:11.918: INFO: rbd-13188 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.918: INFO: rbd-13182 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.926: INFO: rbd-13187 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.927: INFO: rbd-13186 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.940: INFO: rbd-13184 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.955: INFO: rbd-13189 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.971: INFO: rbd-13183 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.971: INFO: rbd-13185 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.972: INFO: rbd-13180 app to be deleted (4 seconds elapsed) Feb 11 11:56:11.989: INFO: rbd-13181 app to be deleted (4 seconds elapsed) Feb 11 11:56:13.918: INFO: rbd-13182 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.918: INFO: rbd-13188 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.925: INFO: rbd-13187 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.927: INFO: Deleting PersistentVolumeClaim rbd-13186 on namespace rbd-1318 Feb 11 11:56:13.936: INFO: waiting for PVC rbd-13186 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:56:13.941: INFO: rbd-13184 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.962: INFO: rbd-13189 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.983: INFO: rbd-13183 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.984: INFO: rbd-13185 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.985: INFO: rbd-13180 app to be deleted (6 seconds elapsed) Feb 11 11:56:13.985: INFO: rbd-13181 app to be deleted (6 seconds elapsed) Feb 11 11:56:15.918: INFO: rbd-13188 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.919: INFO: rbd-13182 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.925: INFO: rbd-13187 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.940: INFO: waiting for PVC rbd-13186 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:56:15.940: INFO: rbd-13184 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.955: INFO: rbd-13189 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.970: INFO: rbd-13185 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.970: INFO: rbd-13183 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.972: INFO: rbd-13180 app to be deleted (8 seconds elapsed) Feb 11 11:56:15.978: INFO: rbd-13181 app to be deleted (8 seconds elapsed) Feb 11 11:56:17.918: INFO: rbd-13188 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.919: INFO: rbd-13182 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.925: INFO: rbd-13187 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.940: INFO: rbd-13184 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.955: INFO: rbd-13189 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.970: INFO: rbd-13183 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.970: INFO: rbd-13185 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.971: INFO: rbd-13180 app to be deleted (10 seconds elapsed) Feb 11 11:56:17.978: INFO: rbd-13181 app to be deleted (10 seconds elapsed) Feb 11 11:56:19.918: INFO: Deleting PersistentVolumeClaim rbd-13182 on namespace rbd-1318 Feb 11 11:56:19.918: INFO: Deleting PersistentVolumeClaim rbd-13188 on namespace rbd-1318 Feb 11 11:56:19.926: INFO: rbd-13187 app to be deleted (12 seconds elapsed) Feb 11 11:56:19.928: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:56:19.934: INFO: waiting for PVC rbd-13182 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:56:19.940: INFO: rbd-13184 app to be deleted (12 seconds elapsed) Feb 11 11:56:19.956: INFO: rbd-13189 app to be deleted (12 seconds elapsed) Feb 11 11:56:19.970: INFO: rbd-13185 app to be deleted (12 seconds elapsed) Feb 11 11:56:19.970: INFO: Deleting PersistentVolumeClaim rbd-13183 on namespace rbd-1318 Feb 11 11:56:19.971: INFO: Deleting PersistentVolumeClaim rbd-13180 on namespace rbd-1318 Feb 11 11:56:19.978: INFO: rbd-13181 app to be deleted (12 seconds elapsed) Feb 11 11:56:19.981: INFO: waiting for PVC rbd-13183 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:56:19.997: INFO: waiting for PVC rbd-13180 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Feb 11 11:56:21.926: INFO: rbd-13187 app to be deleted (14 seconds elapsed) Feb 11 11:56:21.932: INFO: waiting for PVC rbd-13188 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:56:21.937: INFO: waiting for PVC rbd-13182 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:56:21.940: INFO: rbd-13184 app to be deleted (14 seconds elapsed) Feb 11 11:56:21.955: INFO: rbd-13189 app to be deleted (14 seconds elapsed) Feb 11 11:56:21.970: INFO: rbd-13185 app to be deleted (14 seconds elapsed) Feb 11 11:56:21.979: INFO: rbd-13181 app to be deleted (14 seconds elapsed) Feb 11 11:56:21.990: INFO: waiting for PVC rbd-13183 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Feb 11 11:56:22.001: INFO: waiting for PVC rbd-13180 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {