�[1mSTEP�[0m: create rbd snapshots in different pool Jan 20 08:55:41.025: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.234:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-ded5f7a2 --namespace=cephcsi-e2e-ded5f7a2 delete -f -' Jan 20 08:55:41.100: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n" Jan 20 08:55:41.100: INFO: stdout: "volumesnapshotclass.snapshot.storage.k8s.io "csi-rbdplugin-snapclass" deleted\n" Jan 20 08:55:41.106: INFO: ExecWithOptions {Command:[/bin/sh -c ceph osd pool create snapshot-test] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-7tnrm ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Jan 20 08:55:41.106: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:42.578: INFO: stdErr occurred: pool 'snapshot-test' created
Jan 20 08:55:42.585: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-7tnrm ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Jan 20 08:55:42.585: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:43.270: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:43.280: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:55:43.280: INFO: waiting for PVC rbd-pvc (0 seconds elapsed) Jan 20 08:55:45.283: INFO: waiting for PVC rbd-pvc (2 seconds elapsed) Jan 20 08:55:45.287: INFO: Waiting for PV pvc-dc68f8bf-60cc-4ecf-83d1-17c84ab01638 to bind to PVC rbd-pvc Jan 20 08:55:45.287: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound Jan 20 08:55:45.289: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (1.658995ms) Jan 20 08:55:45.289: INFO: Waiting up to 3m0s for PersistentVolume pvc-dc68f8bf-60cc-4ecf-83d1-17c84ab01638 to have phase Bound Jan 20 08:55:45.291: INFO: PersistentVolume pvc-dc68f8bf-60cc-4ecf-83d1-17c84ab01638 found and phase=Bound (1.563997ms) Jan 20 08:55:45.298: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-7tnrm ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Jan 20 08:55:45.298: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.506: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.506: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.506: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.506: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.506: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.506: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.507: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.507: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.506: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.507: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:55:45.514: INFO: snapshot with name rbd-6949 created in rbd-694 namespace Jan 20 08:55:45.514: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6949 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.514: INFO: waiting for snapshot rbd-6949 (0 seconds elapsed) Jan 20 08:55:45.517: INFO: snapshot with name rbd-6941 created in rbd-694 namespace Jan 20 08:55:45.517: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6941 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.517: INFO: waiting for snapshot rbd-6941 (0 seconds elapsed) Jan 20 08:55:45.520: INFO: snapshot with name rbd-6940 created in rbd-694 namespace Jan 20 08:55:45.520: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6940 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.520: INFO: waiting for snapshot rbd-6940 (0 seconds elapsed) Jan 20 08:55:45.520: INFO: snapshot with name rbd-6947 created in rbd-694 namespace Jan 20 08:55:45.520: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6947 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.520: INFO: waiting for snapshot rbd-6947 (0 seconds elapsed) Jan 20 08:55:45.520: INFO: snapshot with name rbd-6945 created in rbd-694 namespace Jan 20 08:55:45.521: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6945 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.521: INFO: waiting for snapshot rbd-6945 (0 seconds elapsed) Jan 20 08:55:45.521: INFO: snapshot with name rbd-6943 created in rbd-694 namespace Jan 20 08:55:45.521: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6943 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.521: INFO: waiting for snapshot rbd-6943 (0 seconds elapsed) Jan 20 08:55:45.521: INFO: snapshot with name rbd-6944 created in rbd-694 namespace Jan 20 08:55:45.521: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6944 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.521: INFO: waiting for snapshot rbd-6944 (0 seconds elapsed) Jan 20 08:55:45.521: INFO: snapshot with name rbd-6942 created in rbd-694 namespace Jan 20 08:55:45.521: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6942 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.521: INFO: waiting for snapshot rbd-6942 (0 seconds elapsed) Jan 20 08:55:45.522: INFO: snapshot with name rbd-6948 created in rbd-694 namespace Jan 20 08:55:45.522: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6948 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.522: INFO: waiting for snapshot rbd-6948 (0 seconds elapsed) Jan 20 08:55:45.529: INFO: snapshot with name rbd-6946 created in rbd-694 namespace Jan 20 08:55:45.529: INFO: waiting for &{{VolumeSnapshot snapshot.storage.k8s.io/v1beta1} {rbd-6946 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []} {{0xc000a18020 } 0xc00111eed0} } to be in ready state Jan 20 08:55:45.529: INFO: waiting for snapshot rbd-6946 (0 seconds elapsed) Jan 20 08:55:47.528: INFO: waiting for snapshot rbd-6949 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6947 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6948 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6943 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6940 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6942 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6941 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6944 (2 seconds elapsed) Jan 20 08:55:47.539: INFO: waiting for snapshot rbd-6945 (2 seconds elapsed) Jan 20 08:55:47.553: INFO: waiting for snapshot rbd-6946 (2 seconds elapsed) Jan 20 08:55:49.528: INFO: waiting for snapshot rbd-6949 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6948 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6943 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6940 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6942 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6947 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6944 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6941 (4 seconds elapsed) Jan 20 08:55:49.539: INFO: waiting for snapshot rbd-6945 (4 seconds elapsed) Jan 20 08:55:49.553: INFO: waiting for snapshot rbd-6946 (4 seconds elapsed) Jan 20 08:55:51.528: INFO: waiting for snapshot rbd-6949 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6947 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6943 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6948 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6942 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6940 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6944 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6941 (6 seconds elapsed) Jan 20 08:55:51.539: INFO: waiting for snapshot rbd-6945 (6 seconds elapsed) Jan 20 08:55:51.543: INFO: snapshot rbd-6944 in false state Jan 20 08:55:51.553: INFO: waiting for snapshot rbd-6946 (6 seconds elapsed) Jan 20 08:55:53.529: INFO: waiting for snapshot rbd-6949 (8 seconds elapsed) Jan 20 08:55:53.539: INFO: waiting for snapshot rbd-6940 (8 seconds elapsed) Jan 20 08:55:53.539: INFO: waiting for snapshot rbd-6948 (8 seconds elapsed) Jan 20 08:55:53.540: INFO: waiting for snapshot rbd-6945 (8 seconds elapsed) Jan 20 08:55:53.540: INFO: waiting for snapshot rbd-6947 (8 seconds elapsed) Jan 20 08:55:53.540: INFO: waiting for snapshot rbd-6943 (8 seconds elapsed) Jan 20 08:55:53.540: INFO: waiting for snapshot rbd-6942 (8 seconds elapsed) Jan 20 08:55:53.540: INFO: waiting for snapshot rbd-6944 (8 seconds elapsed) Jan 20 08:55:53.540: INFO: waiting for snapshot rbd-6941 (8 seconds elapsed) Jan 20 08:55:53.553: INFO: waiting for snapshot rbd-6946 (8 seconds elapsed) Jan 20 08:55:53.572: INFO: snapshot rbd-6947 in false state Jan 20 08:55:53.574: INFO: snapshot rbd-6944 in false state Jan 20 08:55:53.575: INFO: snapshot rbd-6943 in false state Jan 20 08:55:53.575: INFO: snapshot rbd-6945 in false state Jan 20 08:55:53.599: INFO: snapshot rbd-6941 in false state Jan 20 08:55:55.528: INFO: waiting for snapshot rbd-6949 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6948 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6943 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6942 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6947 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6940 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6944 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6941 (10 seconds elapsed) Jan 20 08:55:55.539: INFO: waiting for snapshot rbd-6945 (10 seconds elapsed) Jan 20 08:55:55.542: INFO: snapshot rbd-6948 in false state Jan 20 08:55:55.543: INFO: snapshot rbd-6941 in false state Jan 20 08:55:55.543: INFO: snapshot rbd-6943 in false state Jan 20 08:55:55.543: INFO: snapshot rbd-6940 in false state Jan 20 08:55:55.543: INFO: snapshot rbd-6947 in false state Jan 20 08:55:55.543: INFO: snapshot rbd-6945 in false state Jan 20 08:55:55.543: INFO: snapshot rbd-6944 in false state Jan 20 08:55:55.544: INFO: snapshot rbd-6942 in false state Jan 20 08:55:55.553: INFO: waiting for snapshot rbd-6946 (10 seconds elapsed) Jan 20 08:55:55.555: INFO: snapshot rbd-6946 in false state Jan 20 08:55:57.528: INFO: waiting for snapshot rbd-6949 (12 seconds elapsed) Jan 20 08:55:57.534: INFO: snapshot rbd-6949 in false state Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6947 (12 seconds elapsed) Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6943 (12 seconds elapsed) Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6948 (12 seconds elapsed) Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6940 (12 seconds elapsed) Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6942 (12 seconds elapsed) Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6941 (12 seconds elapsed) Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6944 (12 seconds elapsed) Jan 20 08:55:57.539: INFO: waiting for snapshot rbd-6945 (12 seconds elapsed) Jan 20 08:55:57.543: INFO: snapshot rbd-6945 in false state Jan 20 08:55:57.543: INFO: snapshot rbd-6948 in false state Jan 20 08:55:57.544: INFO: snapshot rbd-6942 in false state Jan 20 08:55:57.544: INFO: snapshot rbd-6940 in false state Jan 20 08:55:57.544: INFO: snapshot rbd-6941 in false state Jan 20 08:55:57.544: INFO: snapshot rbd-6947 in false state Jan 20 08:55:57.544: INFO: snapshot rbd-6943 in false state Jan 20 08:55:57.544: INFO: snapshot rbd-6944 in false state Jan 20 08:55:57.553: INFO: waiting for snapshot rbd-6946 (12 seconds elapsed) Jan 20 08:55:57.556: INFO: snapshot rbd-6946 in false state Jan 20 08:55:59.528: INFO: waiting for snapshot rbd-6949 (14 seconds elapsed) Jan 20 08:55:59.532: INFO: snapshot rbd-6949 in false state Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6948 (14 seconds elapsed) Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6943 (14 seconds elapsed) Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6947 (14 seconds elapsed) Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6942 (14 seconds elapsed) Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6944 (14 seconds elapsed) Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6940 (14 seconds elapsed) Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6941 (14 seconds elapsed) Jan 20 08:55:59.539: INFO: waiting for snapshot rbd-6945 (14 seconds elapsed) Jan 20 08:55:59.546: INFO: snapshot rbd-6943 in false state Jan 20 08:55:59.546: INFO: snapshot rbd-6942 in false state Jan 20 08:55:59.546: INFO: snapshot rbd-6948 in false state Jan 20 08:55:59.546: INFO: snapshot rbd-6947 in false state Jan 20 08:55:59.546: INFO: snapshot rbd-6945 in false state Jan 20 08:55:59.547: INFO: snapshot rbd-6941 in false state Jan 20 08:55:59.547: INFO: snapshot rbd-6944 in false state Jan 20 08:55:59.547: INFO: snapshot rbd-6940 in false state Jan 20 08:55:59.553: INFO: waiting for snapshot rbd-6946 (14 seconds elapsed) Jan 20 08:55:59.555: INFO: snapshot rbd-6946 in false state Jan 20 08:56:01.528: INFO: waiting for snapshot rbd-6949 (16 seconds elapsed) Jan 20 08:56:01.539: INFO: waiting for snapshot rbd-6943 (16 seconds elapsed) Jan 20 08:56:01.539: INFO: waiting for snapshot rbd-6947 (16 seconds elapsed) Jan 20 08:56:01.539: INFO: waiting for snapshot rbd-6940 (16 seconds elapsed) Jan 20 08:56:01.539: INFO: waiting for snapshot rbd-6948 (16 seconds elapsed) Jan 20 08:56:01.539: INFO: waiting for snapshot rbd-6945 (16 seconds elapsed) Jan 20 08:56:01.539: INFO: waiting for snapshot rbd-6942 (16 seconds elapsed) Jan 20 08:56:01.539: INFO: waiting for snapshot rbd-6944 (16 seconds elapsed) Jan 20 08:56:01.540: INFO: waiting for snapshot rbd-6941 (16 seconds elapsed) Jan 20 08:56:01.540: INFO: snapshot rbd-6949 in false state Jan 20 08:56:01.544: INFO: snapshot rbd-6940 in false state Jan 20 08:56:01.544: INFO: snapshot rbd-6945 in false state Jan 20 08:56:01.544: INFO: snapshot rbd-6947 in false state Jan 20 08:56:01.544: INFO: snapshot rbd-6943 in false state Jan 20 08:56:01.544: INFO: snapshot rbd-6944 in false state Jan 20 08:56:01.546: INFO: snapshot rbd-6948 in false state Jan 20 08:56:01.546: INFO: snapshot rbd-6941 in false state Jan 20 08:56:01.546: INFO: snapshot rbd-6942 in false state Jan 20 08:56:01.553: INFO: waiting for snapshot rbd-6946 (16 seconds elapsed) Jan 20 08:56:01.556: INFO: snapshot rbd-6946 in false state Jan 20 08:56:03.528: INFO: waiting for snapshot rbd-6949 (18 seconds elapsed) Jan 20 08:56:03.531: INFO: snapshot rbd-6949 in false state Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6943 (18 seconds elapsed) Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6948 (18 seconds elapsed) Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6940 (18 seconds elapsed) Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6942 (18 seconds elapsed) Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6944 (18 seconds elapsed) Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6947 (18 seconds elapsed) Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6941 (18 seconds elapsed) Jan 20 08:56:03.539: INFO: waiting for snapshot rbd-6945 (18 seconds elapsed) Jan 20 08:56:03.544: INFO: snapshot rbd-6945 in false state Jan 20 08:56:03.544: INFO: snapshot rbd-6944 in false state Jan 20 08:56:03.544: INFO: snapshot rbd-6947 in false state Jan 20 08:56:03.544: INFO: snapshot rbd-6942 in false state Jan 20 08:56:03.544: INFO: snapshot rbd-6943 in false state Jan 20 08:56:03.545: INFO: snapshot rbd-6940 in false state Jan 20 08:56:03.545: INFO: snapshot rbd-6941 in false state Jan 20 08:56:03.545: INFO: snapshot rbd-6948 in false state Jan 20 08:56:03.553: INFO: waiting for snapshot rbd-6946 (18 seconds elapsed) Jan 20 08:56:03.556: INFO: snapshot rbd-6946 in false state Jan 20 08:56:05.528: INFO: waiting for snapshot rbd-6949 (20 seconds elapsed) Jan 20 08:56:05.531: INFO: snapshot rbd-6949 in false state Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6947 (20 seconds elapsed) Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6943 (20 seconds elapsed) Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6948 (20 seconds elapsed) Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6940 (20 seconds elapsed) Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6942 (20 seconds elapsed) Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6941 (20 seconds elapsed) Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6945 (20 seconds elapsed) Jan 20 08:56:05.539: INFO: waiting for snapshot rbd-6944 (20 seconds elapsed) Jan 20 08:56:05.543: INFO: snapshot rbd-6947 in false state Jan 20 08:56:05.543: INFO: snapshot rbd-6943 in false state Jan 20 08:56:05.544: INFO: snapshot rbd-6940 in false state Jan 20 08:56:05.544: INFO: snapshot rbd-6942 in false state Jan 20 08:56:05.544: INFO: snapshot rbd-6948 in false state Jan 20 08:56:05.544: INFO: snapshot rbd-6941 in false state Jan 20 08:56:05.545: INFO: snapshot rbd-6945 in false state Jan 20 08:56:05.545: INFO: snapshot rbd-6944 in false state Jan 20 08:56:05.553: INFO: waiting for snapshot rbd-6946 (20 seconds elapsed) Jan 20 08:56:05.555: INFO: snapshot rbd-6946 in false state Jan 20 08:56:07.528: INFO: waiting for snapshot rbd-6949 (22 seconds elapsed) Jan 20 08:56:07.531: INFO: snapshot rbd-6949 in false state Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6947 (22 seconds elapsed) Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6948 (22 seconds elapsed) Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6943 (22 seconds elapsed) Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6942 (22 seconds elapsed) Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6940 (22 seconds elapsed) Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6944 (22 seconds elapsed) Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6941 (22 seconds elapsed) Jan 20 08:56:07.539: INFO: waiting for snapshot rbd-6945 (22 seconds elapsed) Jan 20 08:56:07.544: INFO: snapshot rbd-6941 in false state Jan 20 08:56:07.544: INFO: snapshot rbd-6948 in false state Jan 20 08:56:07.544: INFO: snapshot rbd-6943 in false state Jan 20 08:56:07.544: INFO: snapshot rbd-6940 in false state Jan 20 08:56:07.544: INFO: snapshot rbd-6944 in false state Jan 20 08:56:07.545: INFO: snapshot rbd-6945 in false state Jan 20 08:56:07.545: INFO: snapshot rbd-6942 in false state Jan 20 08:56:07.546: INFO: snapshot rbd-6947 in false state Jan 20 08:56:07.553: INFO: waiting for snapshot rbd-6946 (22 seconds elapsed) Jan 20 08:56:08.084: INFO: snapshot rbd-6946 in false state Jan 20 08:56:09.528: INFO: waiting for snapshot rbd-6949 (24 seconds elapsed) Jan 20 08:56:09.531: INFO: snapshot rbd-6949 in false state Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6948 (24 seconds elapsed) Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6943 (24 seconds elapsed) Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6947 (24 seconds elapsed) Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6942 (24 seconds elapsed) Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6940 (24 seconds elapsed) Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6944 (24 seconds elapsed) Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6941 (24 seconds elapsed) Jan 20 08:56:09.539: INFO: waiting for snapshot rbd-6945 (24 seconds elapsed) Jan 20 08:56:09.542: INFO: snapshot rbd-6945 in false state Jan 20 08:56:09.542: INFO: snapshot rbd-6943 in false state Jan 20 08:56:09.543: INFO: snapshot rbd-6944 in false state Jan 20 08:56:09.543: INFO: snapshot rbd-6941 in false state Jan 20 08:56:09.543: INFO: snapshot rbd-6947 in false state Jan 20 08:56:09.544: INFO: snapshot rbd-6942 in false state Jan 20 08:56:09.544: INFO: snapshot rbd-6948 in false state Jan 20 08:56:09.544: INFO: snapshot rbd-6940 in false state Jan 20 08:56:09.553: INFO: waiting for snapshot rbd-6946 (24 seconds elapsed) Jan 20 08:56:09.555: INFO: snapshot rbd-6946 in false state Jan 20 08:56:11.528: INFO: waiting for snapshot rbd-6949 (26 seconds elapsed) Jan 20 08:56:11.531: INFO: snapshot rbd-6949 in false state Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6947 (26 seconds elapsed) Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6943 (26 seconds elapsed) Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6948 (26 seconds elapsed) Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6942 (26 seconds elapsed) Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6940 (26 seconds elapsed) Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6944 (26 seconds elapsed) Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6941 (26 seconds elapsed) Jan 20 08:56:11.539: INFO: waiting for snapshot rbd-6945 (26 seconds elapsed) Jan 20 08:56:11.543: INFO: snapshot rbd-6947 in false state Jan 20 08:56:11.543: INFO: snapshot rbd-6942 in false state Jan 20 08:56:11.544: INFO: snapshot rbd-6943 in false state Jan 20 08:56:11.544: INFO: snapshot rbd-6941 in false state Jan 20 08:56:11.544: INFO: snapshot rbd-6940 in false state Jan 20 08:56:11.544: INFO: snapshot rbd-6948 in false state Jan 20 08:56:11.553: INFO: waiting for snapshot rbd-6946 (26 seconds elapsed) Jan 20 08:56:11.556: INFO: snapshot rbd-6946 in false state Jan 20 08:56:13.528: INFO: waiting for snapshot rbd-6949 (28 seconds elapsed) Jan 20 08:56:13.531: INFO: snapshot rbd-6949 in false state Jan 20 08:56:13.539: INFO: waiting for snapshot rbd-6948 (28 seconds elapsed) Jan 20 08:56:13.539: INFO: waiting for snapshot rbd-6947 (28 seconds elapsed) Jan 20 08:56:13.539: INFO: waiting for snapshot rbd-6943 (28 seconds elapsed) Jan 20 08:56:13.539: INFO: waiting for snapshot rbd-6942 (28 seconds elapsed) Jan 20 08:56:13.539: INFO: waiting for snapshot rbd-6941 (28 seconds elapsed) Jan 20 08:56:13.539: INFO: waiting for snapshot rbd-6940 (28 seconds elapsed) Jan 20 08:56:13.544: INFO: snapshot rbd-6947 in false state Jan 20 08:56:13.544: INFO: snapshot rbd-6948 in false state Jan 20 08:56:13.544: INFO: snapshot rbd-6943 in false state Jan 20 08:56:13.553: INFO: waiting for snapshot rbd-6946 (28 seconds elapsed) Jan 20 08:56:13.556: INFO: snapshot rbd-6946 in false state Jan 20 08:56:15.528: INFO: waiting for snapshot rbd-6949 (30 seconds elapsed) Jan 20 08:56:15.539: INFO: waiting for snapshot rbd-6943 (30 seconds elapsed) Jan 20 08:56:15.539: INFO: waiting for snapshot rbd-6947 (30 seconds elapsed) Jan 20 08:56:15.539: INFO: waiting for snapshot rbd-6948 (30 seconds elapsed) Jan 20 08:56:15.553: INFO: waiting for snapshot rbd-6946 (30 seconds elapsed) Jan 20 08:56:15.555: INFO: Deleting PersistentVolumeClaim rbd-pvc on namespace rbd-694 Jan 20 08:56:15.564: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed) Jan 20 08:56:17.566: INFO: waiting for PVC rbd-pvc in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed) Jan 20 08:56:17.573: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=snapshot-test] Namespace:rook-ceph PodName:rook-ceph-tools-5455675849-7tnrm ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Jan 20 08:56:17.573: INFO: >>> kubeConfig: /root/.kube/config Jan 20 08:56:17.783: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6949 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.783: INFO: waiting for PVC rbd-6949 (0 seconds elapsed) Jan 20 08:56:17.788: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6940 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.788: INFO: waiting for PVC rbd-6940 (0 seconds elapsed) Jan 20 08:56:17.795: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6948 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.796: INFO: waiting for PVC rbd-6948 (0 seconds elapsed) Jan 20 08:56:17.795: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6941 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.796: INFO: waiting for PVC rbd-6941 (0 seconds elapsed) Jan 20 08:56:17.795: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6947 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.796: INFO: waiting for PVC rbd-6947 (0 seconds elapsed) Jan 20 08:56:17.802: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6945 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.803: INFO: waiting for PVC rbd-6945 (0 seconds elapsed) Jan 20 08:56:17.807: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6944 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.808: INFO: waiting for PVC rbd-6944 (0 seconds elapsed) Jan 20 08:56:17.808: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6946 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.808: INFO: waiting for PVC rbd-6946 (0 seconds elapsed) Jan 20 08:56:17.808: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6942 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.808: INFO: waiting for PVC rbd-6942 (0 seconds elapsed) Jan 20 08:56:17.807: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-6943 rbd-694 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:*snapshot.storage.k8s.io,Kind:VolumeSnapshot,Name:rbd-6940,},},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state Jan 20 08:56:17.808: INFO: waiting for PVC rbd-6943 (0 seconds elapsed) Jan 20 08:56:19.808: INFO: waiting for PVC rbd-6940 (2 seconds elapsed) Jan 20 08:56:19.808: INFO: waiting for PVC rbd-6949 (2 seconds elapsed) Jan 20 08:56:19.815: INFO: waiting for PVC rbd-6941 (2 seconds elapsed) Jan 20 08:56:19.815: INFO: waiting for PVC rbd-6948 (2 seconds elapsed) Jan 20 08:56:19.815: INFO: waiting for PVC rbd-6947 (2 seconds elapsed) Jan 20 08:56:19.821: INFO: Waiting for PV pvc-be552175-2c18-4cb6-a54c-ce03ed892316 to bind to PVC rbd-6948 Jan 20 08:56:19.821: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6948] to have phase Bound Jan 20 08:56:19.821: INFO: Waiting for PV pvc-45d1adf9-6516-48d0-9183-d407c6ab351e to bind to PVC rbd-6947 Jan 20 08:56:19.821: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6947] to have phase Bound Jan 20 08:56:19.825: INFO: waiting for PVC rbd-6943 (2 seconds elapsed) Jan 20 08:56:19.825: INFO: waiting for PVC rbd-6944 (2 seconds elapsed) Jan 20 08:56:19.825: INFO: waiting for PVC rbd-6946 (2 seconds elapsed) Jan 20 08:56:19.825: INFO: waiting for PVC rbd-6942 (2 seconds elapsed) Jan 20 08:56:19.827: INFO: PersistentVolumeClaim rbd-6948 found and phase=Bound (5.9918ms) Jan 20 08:56:19.827: INFO: Waiting up to 3m0s for PersistentVolume pvc-be552175-2c18-4cb6-a54c-ce03ed892316 to have phase Bound Jan 20 08:56:19.829: INFO: PersistentVolumeClaim rbd-6947 found and phase=Bound (7.806841ms) Jan 20 08:56:19.829: INFO: Waiting up to 3m0s for PersistentVolume pvc-45d1adf9-6516-48d0-9183-d407c6ab351e to have phase Bound Jan 20 08:56:19.830: INFO: waiting for PVC rbd-6945 (2 seconds elapsed) Jan 20 08:56:19.831: INFO: PersistentVolume pvc-be552175-2c18-4cb6-a54c-ce03ed892316 found and phase=Bound (3.727687ms) Jan 20 08:56:19.832: INFO: PersistentVolume pvc-45d1adf9-6516-48d0-9183-d407c6ab351e found and phase=Bound (2.682824ms) Jan 20 08:56:19.838: INFO: Waiting up to rbd-6948 to be in Running state Jan 20 08:56:19.839: INFO: Waiting up to rbd-6947 to be in Running state Jan 20 08:56:19.842: INFO: rbd-6948 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:19.842: INFO: rbd-6947 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:21.808: INFO: waiting for PVC rbd-6940 (4 seconds elapsed) Jan 20 08:56:21.808: INFO: waiting for PVC rbd-6949 (4 seconds elapsed) Jan 20 08:56:21.812: INFO: Waiting for PV pvc-dd37aeb8-698a-468a-aff8-dc577504bf97 to bind to PVC rbd-6949 Jan 20 08:56:21.812: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6949] to have phase Bound Jan 20 08:56:21.813: INFO: Waiting for PV pvc-e0f6acf7-c92f-4e2a-9bd9-951aab279178 to bind to PVC rbd-6940 Jan 20 08:56:21.813: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6940] to have phase Bound Jan 20 08:56:21.814: INFO: PersistentVolumeClaim rbd-6949 found and phase=Bound (1.77617ms) Jan 20 08:56:21.814: INFO: Waiting up to 3m0s for PersistentVolume pvc-dd37aeb8-698a-468a-aff8-dc577504bf97 to have phase Bound Jan 20 08:56:21.815: INFO: waiting for PVC rbd-6941 (4 seconds elapsed) Jan 20 08:56:21.815: INFO: PersistentVolumeClaim rbd-6940 found and phase=Bound (1.808591ms) Jan 20 08:56:21.815: INFO: Waiting up to 3m0s for PersistentVolume pvc-e0f6acf7-c92f-4e2a-9bd9-951aab279178 to have phase Bound Jan 20 08:56:21.816: INFO: PersistentVolume pvc-dd37aeb8-698a-468a-aff8-dc577504bf97 found and phase=Bound (2.206862ms) Jan 20 08:56:21.817: INFO: PersistentVolume pvc-e0f6acf7-c92f-4e2a-9bd9-951aab279178 found and phase=Bound (1.823381ms) Jan 20 08:56:21.819: INFO: Waiting for PV pvc-c9fc01dd-ecfb-43c5-a9b7-ee52d7f1dab9 to bind to PVC rbd-6941 Jan 20 08:56:21.819: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6941] to have phase Bound Jan 20 08:56:21.821: INFO: PersistentVolumeClaim rbd-6941 found and phase=Bound (1.910936ms) Jan 20 08:56:21.821: INFO: Waiting up to 3m0s for PersistentVolume pvc-c9fc01dd-ecfb-43c5-a9b7-ee52d7f1dab9 to have phase Bound Jan 20 08:56:21.823: INFO: PersistentVolume pvc-c9fc01dd-ecfb-43c5-a9b7-ee52d7f1dab9 found and phase=Bound (2.060414ms) Jan 20 08:56:21.825: INFO: Waiting up to rbd-6940 to be in Running state Jan 20 08:56:21.825: INFO: waiting for PVC rbd-6942 (4 seconds elapsed) Jan 20 08:56:21.825: INFO: waiting for PVC rbd-6946 (4 seconds elapsed) Jan 20 08:56:21.825: INFO: waiting for PVC rbd-6944 (4 seconds elapsed) Jan 20 08:56:21.825: INFO: waiting for PVC rbd-6943 (4 seconds elapsed) Jan 20 08:56:21.828: INFO: Waiting up to rbd-6949 to be in Running state Jan 20 08:56:21.828: INFO: rbd-6940 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:21.830: INFO: waiting for PVC rbd-6945 (4 seconds elapsed) Jan 20 08:56:21.837: INFO: rbd-6949 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:21.837: INFO: Waiting for PV pvc-353465ab-12c6-4d8c-bf94-7f241b9025e0 to bind to PVC rbd-6944 Jan 20 08:56:21.837: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6944] to have phase Bound Jan 20 08:56:21.846: INFO: PersistentVolumeClaim rbd-6944 found and phase=Bound (8.72677ms) Jan 20 08:56:21.846: INFO: Waiting up to 3m0s for PersistentVolume pvc-353465ab-12c6-4d8c-bf94-7f241b9025e0 to have phase Bound Jan 20 08:56:21.847: INFO: Waiting up to rbd-6941 to be in Running state Jan 20 08:56:21.855: INFO: rbd-6948 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:21.858: INFO: rbd-6947 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:21.859: INFO: rbd-6941 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:21.859: INFO: PersistentVolume pvc-353465ab-12c6-4d8c-bf94-7f241b9025e0 found and phase=Bound (12.562368ms) Jan 20 08:56:21.884: INFO: Waiting up to rbd-6944 to be in Running state Jan 20 08:56:21.889: INFO: rbd-6944 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:23.825: INFO: waiting for PVC rbd-6943 (6 seconds elapsed) Jan 20 08:56:23.825: INFO: waiting for PVC rbd-6946 (6 seconds elapsed) Jan 20 08:56:23.825: INFO: waiting for PVC rbd-6942 (6 seconds elapsed) Jan 20 08:56:23.830: INFO: rbd-6940 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:23.830: INFO: waiting for PVC rbd-6945 (6 seconds elapsed) Jan 20 08:56:23.830: INFO: Waiting for PV pvc-2681746e-ddc6-497e-bdf7-bf47c028bc72 to bind to PVC rbd-6943 Jan 20 08:56:23.830: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6943] to have phase Bound Jan 20 08:56:23.830: INFO: Waiting for PV pvc-eb95182b-87b6-440a-9765-6522f6905159 to bind to PVC rbd-6942 Jan 20 08:56:23.830: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6942] to have phase Bound Jan 20 08:56:23.830: INFO: Waiting for PV pvc-e15be9ce-960c-491f-b12d-bdc8ed78739a to bind to PVC rbd-6946 Jan 20 08:56:23.830: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6946] to have phase Bound Jan 20 08:56:23.832: INFO: PersistentVolumeClaim rbd-6942 found and phase=Bound (1.870368ms) Jan 20 08:56:23.832: INFO: Waiting up to 3m0s for PersistentVolume pvc-eb95182b-87b6-440a-9765-6522f6905159 to have phase Bound Jan 20 08:56:23.832: INFO: PersistentVolumeClaim rbd-6946 found and phase=Bound (2.117748ms) Jan 20 08:56:23.832: INFO: Waiting up to 3m0s for PersistentVolume pvc-e15be9ce-960c-491f-b12d-bdc8ed78739a to have phase Bound Jan 20 08:56:23.832: INFO: PersistentVolumeClaim rbd-6943 found and phase=Bound (2.549026ms) Jan 20 08:56:23.833: INFO: Waiting up to 3m0s for PersistentVolume pvc-2681746e-ddc6-497e-bdf7-bf47c028bc72 to have phase Bound Jan 20 08:56:23.835: INFO: PersistentVolume pvc-eb95182b-87b6-440a-9765-6522f6905159 found and phase=Bound (2.460962ms) Jan 20 08:56:23.835: INFO: Waiting for PV pvc-3320890c-5acb-431e-881f-19cb91ea6b81 to bind to PVC rbd-6945 Jan 20 08:56:23.835: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-6945] to have phase Bound Jan 20 08:56:23.835: INFO: PersistentVolume pvc-e15be9ce-960c-491f-b12d-bdc8ed78739a found and phase=Bound (2.894621ms) Jan 20 08:56:23.835: INFO: PersistentVolume pvc-2681746e-ddc6-497e-bdf7-bf47c028bc72 found and phase=Bound (2.927064ms) Jan 20 08:56:23.838: INFO: PersistentVolumeClaim rbd-6945 found and phase=Bound (3.075746ms) Jan 20 08:56:23.838: INFO: Waiting up to 3m0s for PersistentVolume pvc-3320890c-5acb-431e-881f-19cb91ea6b81 to have phase Bound Jan 20 08:56:23.841: INFO: PersistentVolume pvc-3320890c-5acb-431e-881f-19cb91ea6b81 found and phase=Bound (2.707445ms) Jan 20 08:56:23.841: INFO: rbd-6949 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:23.843: INFO: Waiting up to rbd-6942 to be in Running state Jan 20 08:56:23.849: INFO: Waiting up to rbd-6946 to be in Running state Jan 20 08:56:23.853: INFO: Waiting up to rbd-6943 to be in Running state Jan 20 08:56:23.855: INFO: rbd-6948 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:23.855: INFO: rbd-6946 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:23.858: INFO: rbd-6947 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:23.860: INFO: rbd-6942 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:23.862: INFO: rbd-6943 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:23.865: INFO: Waiting up to rbd-6945 to be in Running state Jan 20 08:56:23.865: INFO: rbd-6941 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:23.868: INFO: rbd-6945 app is in Pending phase expected to be in Running state (0 seconds elapsed) Jan 20 08:56:23.894: INFO: rbd-6944 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:25.835: INFO: rbd-6940 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:25.840: INFO: rbd-6949 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:25.844: INFO: rbd-6947 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:25.845: INFO: rbd-6948 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:25.857: INFO: rbd-6946 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:25.861: INFO: rbd-6941 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:25.862: INFO: rbd-6942 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:25.866: INFO: rbd-6943 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:25.870: INFO: rbd-6945 app is in Pending phase expected to be in Running state (2 seconds elapsed) Jan 20 08:56:25.892: INFO: rbd-6944 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:27.831: INFO: rbd-6940 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:27.839: INFO: rbd-6949 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:27.843: INFO: rbd-6947 app is in Pending phase expected to be in Running state (8 seconds elapsed) Jan 20 08:56:27.844: INFO: rbd-6948 app is in Pending phase expected to be in Running state (8 seconds elapsed) Jan 20 08:56:27.857: INFO: rbd-6946 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:27.860: INFO: rbd-6941 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:27.862: INFO: rbd-6942 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:27.865: INFO: rbd-6943 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:27.870: INFO: rbd-6945 app is in Pending phase expected to be in Running state (4 seconds elapsed) Jan 20 08:56:27.892: INFO: rbd-6944 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:29.830: INFO: rbd-6940 app is in Pending phase expected to be in Running state (8 seconds elapsed) Jan 20 08:56:29.839: INFO: rbd-6949 app is in Pending phase expected to be in Running state (8 seconds elapsed) Jan 20 08:56:29.844: INFO: rbd-6947 app is in Pending phase expected to be in Running state (10 seconds elapsed) Jan 20 08:56:29.844: INFO: rbd-6948 app is in Pending phase expected to be in Running state (10 seconds elapsed) Jan 20 08:56:29.857: INFO: rbd-6946 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:29.861: INFO: rbd-6941 app is in Pending phase expected to be in Running state (8 seconds elapsed) Jan 20 08:56:29.862: INFO: rbd-6942 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:29.864: INFO: rbd-6943 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:29.870: INFO: rbd-6945 app is in Pending phase expected to be in Running state (6 seconds elapsed) Jan 20 08:56:29.891: INFO: rbd-6944 app is in Pending phase expected to be in Running state (8 seconds elapsed) Jan 20 08:56:31.830: INFO: rbd-6940 app is in Pending phase expected to be in Running state (10 seconds elapsed) Jan 20 08:56:31.839: INFO: rbd-6949 app is in Pending phase expected to be in Running state (10 seconds elapsed) Jan 20 08:56:31.844: INFO: rbd-6948 app is in Pending phase expected to be in Running state (12 seconds elapsed) Jan 20 08:56:31.844: INFO: rbd-6947 app is in Pending phase expected to be in Running state (12 seconds elapsed)
Jan 20 09:05:45.870: INFO: rbd-6945 app is in Pending phase expected to be in Running state (562 seconds elapsed) Jan 20 09:05:45.892: INFO: rbd-6944 app is in Pending phase expected to be in Running state (564 seconds elapsed) Jan 20 09:05:47.830: INFO: rbd-6940 app is in Pending phase expected to be in Running state (566 seconds elapsed) Jan 20 09:05:47.840: INFO: rbd-6949 app is in Pending phase expected to be in Running state (566 seconds elapsed) Jan 20 09:05:47.844: INFO: rbd-6948 app is in Pending phase expected to be in Running state (568 seconds elapsed) Jan 20 09:05:47.844: INFO: rbd-6947 app is in Pending phase expected to be in Running state (568 seconds elapsed) Jan 20 09:05:47.857: INFO: rbd-6946 app is in Pending phase expected to be in Running state (564 seconds elapsed) Jan 20 09:05:47.861: INFO: rbd-6941 app is in Pending phase expected to be in Running state (566 seconds elapsed) Jan 20 09:05:47.862: INFO: rbd-6942 app is in Pending phase expected to be in Running state (564 seconds elapsed) Jan 20 09:05:47.864: INFO: rbd-6943 app is in Pending phase expected to be in Running state (564 seconds elapsed) Jan 20 09:05:47.870: INFO: rbd-6945 app is in Pending phase expected to be in Running state (564 seconds elapsed) Jan 20 09:05:47.892: INFO: rbd-6944 app is in Pending phase expected to be in Running state (566 seconds elapsed) Jan 20 09:05:49.831: INFO: rbd-6940 app is in Pending phase expected to be in Running state (568 seconds elapsed) Jan 20 09:05:49.839: INFO: rbd-6949 app is in Pending phase expected to be in Running state (568 seconds elapsed) Jan 20 09:05:49.844: INFO: rbd-6947 app is in Pending phase expected to be in Running state (570 seconds elapsed) Jan 20 09:05:49.844: INFO: rbd-6948 app is in Pending phase expected to be in Running state (570 seconds elapsed) Jan 20 09:05:49.857: INFO: rbd-6946 app is in Pending phase expected to be in Running state (566 seconds elapsed) Jan 20 09:05:49.860: INFO: rbd-6941 app is in Pending phase expected to be in Running state (568 seconds elapsed) Jan 20 09:05:49.862: INFO: rbd-6942 app is in Pending phase expected to be in Running state (566 seconds elapsed) Jan 20 09:05:49.864: INFO: rbd-6943 app is in Pending phase expected to be in Running state (566 seconds elapsed)
Jan 20 09:06:23.859: INFO: rbd-6946 app is in Pending phase expected to be in Running state (600 seconds elapsed) Jan 20 09:06:23.863: INFO: rbd-6942 app is in Pending phase expected to be in Running state (600 seconds elapsed) Jan 20 09:06:23.865: INFO: rbd-6942 app is in Pending phase expected to be in Running state (600 seconds elapsed) Jan 20 09:06:23.865: INFO: rbd-6943 app is in Pending phase expected to be in Running state (600 seconds elapsed) Jan 20 09:06:23.867: INFO: rbd-6943 app is in Pending phase expected to be in Running state (600 seconds elapsed) Jan 20 09:06:23.869: INFO: rbd-6945 app is in Pending phase expected to be in Running state (600 seconds elapsed) Jan 20 09:06:23.871: INFO: rbd-6945 app is in Pending phase expected to be in Running state (600 seconds elapsed) Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6940): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6941): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6942): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6943): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6944): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6945): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6946): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6947): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6948): timed out waiting for the condition Jan 20 09:06:23.871: INFO: failed to create PVC and application (rbd-6949): timed out waiting for the condition Jan 20 09:06:23.871: FAIL: failed to validate snapshot in different pool with error creating PVCs and applications failed, 10 errors were logged