gistfile1.txt

2023-07-18 22:12:02.421991 D | exec: Running command: ceph status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-07-18 22:12:03.028288 I | clusterdisruption-controller: osd is down in failure domain "c10-z1-sea-pahadi-net" and pgs are not active+clean. pg health: "cluster is not fully clean. PGs: [{StateName:unknown Count:186} {StateName:down Count:9} {StateName:incomplete Count:1}]"
2023-07-18 22:12:03.028336 D | exec: Running command: ceph osd dump --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-07-18 22:12:03.436202 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.437378 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.438856 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.439949 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.440975 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.442049 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.443150 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.444278 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.445443 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.446518 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.447547 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.448742 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:03.448781 D | clusterdisruption-controller: deleting default pdb with maxUnavailable=1 for all osd
2023-07-18 22:12:03.449808 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:07.437661 D | clusterdisruption-controller: reconciling "rook-ceph/rook-ceph"
2023-07-18 22:12:07.442114 D | clusterdisruption-controller: osd "6" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442134 I | clusterdisruption-controller: osd "rook-ceph-osd-6" is down and a possible node drain is detected
2023-07-18 22:12:07.442181 D | clusterdisruption-controller: osd "7" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442189 I | clusterdisruption-controller: osd "rook-ceph-osd-7" is down and a possible node drain is detected
2023-07-18 22:12:07.442229 D | clusterdisruption-controller: osd "45" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442240 I | clusterdisruption-controller: osd "rook-ceph-osd-45" is down and a possible node drain is detected
2023-07-18 22:12:07.442282 D | clusterdisruption-controller: osd "50" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442292 I | clusterdisruption-controller: osd "rook-ceph-osd-50" is down and a possible node drain is detected
2023-07-18 22:12:07.442331 D | clusterdisruption-controller: osd "34" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442340 I | clusterdisruption-controller: osd "rook-ceph-osd-34" is down and a possible node drain is detected
2023-07-18 22:12:07.442377 D | clusterdisruption-controller: osd "40" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442386 I | clusterdisruption-controller: osd "rook-ceph-osd-40" is down and a possible node drain is detected
2023-07-18 22:12:07.442425 D | clusterdisruption-controller: osd "23" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442434 I | clusterdisruption-controller: osd "rook-ceph-osd-23" is down and a possible node drain is detected
2023-07-18 22:12:07.442471 D | clusterdisruption-controller: osd "28" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442480 I | clusterdisruption-controller: osd "rook-ceph-osd-28" is down and a possible node drain is detected
2023-07-18 22:12:07.442517 D | clusterdisruption-controller: osd "32" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442527 I | clusterdisruption-controller: osd "rook-ceph-osd-32" is down and a possible node drain is detected
2023-07-18 22:12:07.442564 D | clusterdisruption-controller: osd "2" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442573 I | clusterdisruption-controller: osd "rook-ceph-osd-2" is down and a possible node drain is detected
2023-07-18 22:12:07.442613 D | clusterdisruption-controller: osd "22" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442622 I | clusterdisruption-controller: osd "rook-ceph-osd-22" is down and a possible node drain is detected
2023-07-18 22:12:07.442658 D | clusterdisruption-controller: osd "21" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442668 I | clusterdisruption-controller: osd "rook-ceph-osd-21" is down and a possible node drain is detected
2023-07-18 22:12:07.442703 D | clusterdisruption-controller: osd "25" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442713 I | clusterdisruption-controller: osd "rook-ceph-osd-25" is down and a possible node drain is detected
2023-07-18 22:12:07.442747 D | clusterdisruption-controller: osd "30" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442756 I | clusterdisruption-controller: osd "rook-ceph-osd-30" is down and a possible node drain is detected
2023-07-18 22:12:07.442790 D | clusterdisruption-controller: osd "37" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442800 I | clusterdisruption-controller: osd "rook-ceph-osd-37" is down and a possible node drain is detected
2023-07-18 22:12:07.442835 D | clusterdisruption-controller: osd "46" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442845 I | clusterdisruption-controller: osd "rook-ceph-osd-46" is down and a possible node drain is detected
2023-07-18 22:12:07.442878 D | clusterdisruption-controller: osd "17" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442888 I | clusterdisruption-controller: osd "rook-ceph-osd-17" is down and a possible node drain is detected
2023-07-18 22:12:07.442939 D | clusterdisruption-controller: osd "18" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442949 I | clusterdisruption-controller: osd "rook-ceph-osd-18" is down and a possible node drain is detected
2023-07-18 22:12:07.442988 D | clusterdisruption-controller: osd "39" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.442997 I | clusterdisruption-controller: osd "rook-ceph-osd-39" is down and a possible node drain is detected
2023-07-18 22:12:07.443035 D | clusterdisruption-controller: osd "43" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443045 I | clusterdisruption-controller: osd "rook-ceph-osd-43" is down and a possible node drain is detected
2023-07-18 22:12:07.443081 D | clusterdisruption-controller: osd "24" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443091 I | clusterdisruption-controller: osd "rook-ceph-osd-24" is down and a possible node drain is detected
2023-07-18 22:12:07.443126 D | clusterdisruption-controller: osd "36" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443135 I | clusterdisruption-controller: osd "rook-ceph-osd-36" is down and a possible node drain is detected
2023-07-18 22:12:07.443171 D | clusterdisruption-controller: osd "16" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443181 I | clusterdisruption-controller: osd "rook-ceph-osd-16" is down and a possible node drain is detected
2023-07-18 22:12:07.443216 D | clusterdisruption-controller: osd "4" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443225 I | clusterdisruption-controller: osd "rook-ceph-osd-4" is down and a possible node drain is detected
2023-07-18 22:12:07.443458 I | clusterdisruption-controller: osd "rook-ceph-osd-52" is down but no node drain is detected
2023-07-18 22:12:07.443510 D | clusterdisruption-controller: osd "14" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443520 I | clusterdisruption-controller: osd "rook-ceph-osd-14" is down and a possible node drain is detected
2023-07-18 22:12:07.443556 D | clusterdisruption-controller: osd "29" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443565 I | clusterdisruption-controller: osd "rook-ceph-osd-29" is down and a possible node drain is detected
2023-07-18 22:12:07.443602 D | clusterdisruption-controller: osd "8" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443612 I | clusterdisruption-controller: osd "rook-ceph-osd-8" is down and a possible node drain is detected
2023-07-18 22:12:07.443648 D | clusterdisruption-controller: osd "15" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443657 I | clusterdisruption-controller: osd "rook-ceph-osd-15" is down and a possible node drain is detected
2023-07-18 22:12:07.443693 D | clusterdisruption-controller: osd "48" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443703 I | clusterdisruption-controller: osd "rook-ceph-osd-48" is down and a possible node drain is detected
2023-07-18 22:12:07.443740 D | clusterdisruption-controller: osd "33" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443750 I | clusterdisruption-controller: osd "rook-ceph-osd-33" is down and a possible node drain is detected
2023-07-18 22:12:07.443785 D | clusterdisruption-controller: osd "35" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443795 I | clusterdisruption-controller: osd "rook-ceph-osd-35" is down and a possible node drain is detected
2023-07-18 22:12:07.443831 D | clusterdisruption-controller: osd "10" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443840 I | clusterdisruption-controller: osd "rook-ceph-osd-10" is down and a possible node drain is detected
2023-07-18 22:12:07.443876 D | clusterdisruption-controller: osd "19" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443886 I | clusterdisruption-controller: osd "rook-ceph-osd-19" is down and a possible node drain is detected
2023-07-18 22:12:07.443920 D | clusterdisruption-controller: osd "38" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443930 I | clusterdisruption-controller: osd "rook-ceph-osd-38" is down and a possible node drain is detected
2023-07-18 22:12:07.443965 D | clusterdisruption-controller: osd "0" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.443975 I | clusterdisruption-controller: osd "rook-ceph-osd-0" is down and a possible node drain is detected
2023-07-18 22:12:07.444013 D | clusterdisruption-controller: osd "11" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444022 I | clusterdisruption-controller: osd "rook-ceph-osd-11" is down and a possible node drain is detected
2023-07-18 22:12:07.444063 D | clusterdisruption-controller: osd "9" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444072 I | clusterdisruption-controller: osd "rook-ceph-osd-9" is down and a possible node drain is detected
2023-07-18 22:12:07.444106 D | clusterdisruption-controller: osd "20" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444117 I | clusterdisruption-controller: osd "rook-ceph-osd-20" is down and a possible node drain is detected
2023-07-18 22:12:07.444154 D | clusterdisruption-controller: osd "42" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444164 I | clusterdisruption-controller: osd "rook-ceph-osd-42" is down and a possible node drain is detected
2023-07-18 22:12:07.444202 D | clusterdisruption-controller: osd "47" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444212 I | clusterdisruption-controller: osd "rook-ceph-osd-47" is down and a possible node drain is detected
2023-07-18 22:12:07.444246 D | clusterdisruption-controller: osd "5" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444255 I | clusterdisruption-controller: osd "rook-ceph-osd-5" is down and a possible node drain is detected
2023-07-18 22:12:07.444291 D | clusterdisruption-controller: osd "13" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444301 I | clusterdisruption-controller: osd "rook-ceph-osd-13" is down and a possible node drain is detected
2023-07-18 22:12:07.444338 D | clusterdisruption-controller: osd "27" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444347 I | clusterdisruption-controller: osd "rook-ceph-osd-27" is down and a possible node drain is detected
2023-07-18 22:12:07.444381 D | clusterdisruption-controller: osd "49" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444392 I | clusterdisruption-controller: osd "rook-ceph-osd-49" is down and a possible node drain is detected
2023-07-18 22:12:07.444426 D | clusterdisruption-controller: osd "51" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444436 I | clusterdisruption-controller: osd "rook-ceph-osd-51" is down and a possible node drain is detected
2023-07-18 22:12:07.444472 D | clusterdisruption-controller: osd "1" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444481 I | clusterdisruption-controller: osd "rook-ceph-osd-1" is down and a possible node drain is detected
2023-07-18 22:12:07.444519 D | clusterdisruption-controller: osd "26" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444528 I | clusterdisruption-controller: osd "rook-ceph-osd-26" is down and a possible node drain is detected
2023-07-18 22:12:07.444562 D | clusterdisruption-controller: osd "31" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444571 I | clusterdisruption-controller: osd "rook-ceph-osd-31" is down and a possible node drain is detected
2023-07-18 22:12:07.444606 D | clusterdisruption-controller: osd "41" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444615 I | clusterdisruption-controller: osd "rook-ceph-osd-41" is down and a possible node drain is detected
2023-07-18 22:12:07.444650 D | clusterdisruption-controller: osd "44" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444660 I | clusterdisruption-controller: osd "rook-ceph-osd-44" is down and a possible node drain is detected
2023-07-18 22:12:07.444696 D | clusterdisruption-controller: osd "12" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444706 I | clusterdisruption-controller: osd "rook-ceph-osd-12" is down and a possible node drain is detected
2023-07-18 22:12:07.444739 D | clusterdisruption-controller: osd "3" POD is not assigned to any node. assuming node drain
2023-07-18 22:12:07.444748 I | clusterdisruption-controller: osd "rook-ceph-osd-3" is down and a possible node drain is detected
2023-07-18 22:12:07.444811 D | exec: Running command: ceph status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-07-18 22:12:08.073595 I | clusterdisruption-controller: osd is down in failure domain "c10-z1-sea-pahadi-net" and pgs are not active+clean. pg health: "cluster is not fully clean. PGs: [{StateName:unknown Count:186} {StateName:down Count:9} {StateName:incomplete Count:1}]"
2023-07-18 22:12:08.073649 D | exec: Running command: ceph osd dump --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-07-18 22:12:08.494292 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.495141 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.496170 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.497362 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.498387 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.499511 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.500570 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.501709 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.502717 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.503772 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.504896 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.505973 D | op-k8sutil: kubernetes version fetched 1.27.2
2023-07-18 22:12:08.506016 D | clusterdisruption-controller: deleting default pdb with maxUnavailable=1 for all osd
2023-07-18 22:12:08.507104 D | op-k8sutil: kubernetes version fetched 1.27.2