Created
September 22, 2021 12:40
-
-
Save ShyamsundarR/2041ac292ab57dd6f2e01b60452f12af to your computer and use it in GitHub Desktop.
Log output that demonstrates Ceph RBD image promotion/demotion across clusters, and results in an image leak
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ ./rbd-image-leak.sh | |
| === Creating image EAST === | |
| === Enabling mirror EAST === | |
| Mirroring enabled | |
| === Listing snapshot schedule EAST === | |
| every 2m starting at 14:00:00-05:00 | |
| === Mirror image status EAST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+stopped | |
| description: local image is primary | |
| service: a on east | |
| last_update: 2021-09-22 12:10:26 | |
| peer_sites: | |
| name: 4cf1bc03-3bd6-470c-8de7-7351205b3588 | |
| state: down+unknown | |
| description: status not found | |
| last_update: | |
| snapshots: | |
| 13308 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.01ae7cf1-e456-4421-96fa-4f18a769380e (peer_uuids:[3ab0dc51-3bbf-4440-8b19-0d9fe64818c2]) | |
| === Mirror image status WEST === | |
| rbd: mirroring not enabled on the image | |
| command terminated with exit code 22 | |
| === Perform IO on EAST === | |
| bench type write io_size 4096 io_threads 2 bytes 15728640 pattern sequential | |
| SEC OPS OPS/SEC BYTES/SEC | |
| elapsed: 11 ops: 3840 ops/sec: 333.768 bytes/sec: 1.3 MiB/s | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Force promote image on WEST === | |
| Image promoted to primary | |
| === Perform IO on WEST === | |
| bench type write io_size 4096 io_threads 2 bytes 15728640 pattern sequential | |
| SEC OPS OPS/SEC BYTES/SEC | |
| elapsed: 3 ops: 3840 ops/sec: 1023.45 bytes/sec: 4.0 MiB/s | |
| === Perform IO on EAST === | |
| bench type write io_size 4096 io_threads 2 bytes 15728640 pattern sequential | |
| SEC OPS OPS/SEC BYTES/SEC | |
| elapsed: 3 ops: 3840 ops/sec: 1042.35 bytes/sec: 4.1 MiB/s | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Mirror image status EAST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+stopped | |
| description: local image is primary | |
| service: a on east | |
| last_update: 2021-09-22 12:15:54 | |
| peer_sites: | |
| name: 4cf1bc03-3bd6-470c-8de7-7351205b3588 | |
| state: up+stopped | |
| description: local image is primary | |
| last_update: 2021-09-22 12:15:33 | |
| snapshots: | |
| 13309 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.e03b65f0-20fc-40f8-b711-4bf0528e1137 (peer_uuids:[3ab0dc51-3bbf-4440-8b19-0d9fe64818c2]) | |
| 13311 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.b909ee3e-0735-4448-a95c-50aef9ce23b8 (peer_uuids:[3ab0dc51-3bbf-4440-8b19-0d9fe64818c2]) | |
| 13312 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.886639ec-1b0d-40fb-99f8-79b6fc661e78 (peer_uuids:[3ab0dc51-3bbf-4440-8b19-0d9fe64818c2]) | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+stopped | |
| description: local image is primary | |
| service: a on west | |
| last_update: 2021-09-22 12:16:19 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+stopped | |
| description: local image is primary | |
| last_update: 2021-09-22 12:16:10 | |
| snapshots: | |
| 13097 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.503517d4-4c8a-4099-917e-3cf2b5169e14 (peer_uuids:[b00051bd-b7f4-436b-9755-d2c2ca285f5e]) | |
| === Demoting EAST === | |
| Image demoted to non-primary | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Mirror image status EAST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+error | |
| description: split-brain | |
| service: a on east | |
| last_update: 2021-09-22 12:18:24 | |
| peer_sites: | |
| name: 4cf1bc03-3bd6-470c-8de7-7351205b3588 | |
| state: up+stopped | |
| description: local image is primary | |
| last_update: 2021-09-22 12:18:33 | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+stopped | |
| description: local image is primary | |
| service: a on west | |
| last_update: 2021-09-22 12:18:49 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+error | |
| description: split-brain | |
| last_update: 2021-09-22 12:18:40 | |
| snapshots: | |
| 13097 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.503517d4-4c8a-4099-917e-3cf2b5169e14 (peer_uuids:[b00051bd-b7f4-436b-9755-d2c2ca285f5e]) | |
| === Resync EAST === | |
| Flagged image for resync from primary | |
| === Sleeping for 150 seconds for resync catch up === | |
| === Mirror image status EAST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+replaying | |
| description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":15728640.0,"local_snapshot_timestamp":1632312811,"remote_snapshot_timestamp":1632312811,"replay_state":"idle"} | |
| service: a on east | |
| last_update: 2021-09-22 12:20:54 | |
| peer_sites: | |
| name: 4cf1bc03-3bd6-470c-8de7-7351205b3588 | |
| state: up+stopped | |
| description: local image is primary | |
| last_update: 2021-09-22 12:21:03 | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+stopped | |
| description: local image is primary | |
| service: a on west | |
| last_update: 2021-09-22 12:21:19 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+replaying | |
| description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":15728640.0,"local_snapshot_timestamp":1632312811,"remote_snapshot_timestamp":1632312811,"replay_state":"idle"} | |
| last_update: 2021-09-22 12:21:09 | |
| snapshots: | |
| 13097 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.503517d4-4c8a-4099-917e-3cf2b5169e14 (peer_uuids:[b00051bd-b7f4-436b-9755-d2c2ca285f5e]) | |
| === Perform IO on WEST === | |
| bench type write io_size 4096 io_threads 2 bytes 15728640 pattern sequential | |
| SEC OPS OPS/SEC BYTES/SEC | |
| elapsed: 3 ops: 3840 ops/sec: 1105.04 bytes/sec: 4.3 MiB/s | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Mirror image status EAST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+replaying | |
| description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":15728640.0,"local_snapshot_timestamp":1632312811,"remote_snapshot_timestamp":1632312811,"replay_state":"idle"} | |
| service: a on east | |
| last_update: 2021-09-22 12:23:24 | |
| peer_sites: | |
| name: 4cf1bc03-3bd6-470c-8de7-7351205b3588 | |
| state: up+stopped | |
| description: local image is primary | |
| last_update: 2021-09-22 12:23:33 | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+stopped | |
| description: local image is primary | |
| service: a on west | |
| last_update: 2021-09-22 12:23:49 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+replaying | |
| description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":15728640.0,"local_snapshot_timestamp":1632312811,"remote_snapshot_timestamp":1632312811,"replay_state":"idle"} | |
| last_update: 2021-09-22 12:24:09 | |
| snapshots: | |
| 13097 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.503517d4-4c8a-4099-917e-3cf2b5169e14 (peer_uuids:[b00051bd-b7f4-436b-9755-d2c2ca285f5e]) | |
| === Demote on WEST === | |
| Image demoted to non-primary | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Mirror image status EAST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+unknown | |
| description: remote image demoted | |
| service: a on east | |
| last_update: 2021-09-22 12:26:25 | |
| peer_sites: | |
| name: 4cf1bc03-3bd6-470c-8de7-7351205b3588 | |
| state: up+unknown | |
| description: remote image demoted | |
| last_update: 2021-09-22 12:26:34 | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+unknown | |
| description: remote image demoted | |
| service: a on west | |
| last_update: 2021-09-22 12:26:50 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+unknown | |
| description: remote image demoted | |
| last_update: 2021-09-22 12:26:41 | |
| === Promote on EAST === | |
| Image promoted to primary | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Mirror image status EAST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+stopped | |
| description: local image is primary | |
| service: a on east | |
| last_update: 2021-09-22 12:28:54 | |
| peer_sites: | |
| name: 4cf1bc03-3bd6-470c-8de7-7351205b3588 | |
| state: up+replaying | |
| description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":15728640.0,"local_snapshot_timestamp":1632313598,"remote_snapshot_timestamp":1632313598,"replay_state":"idle"} | |
| last_update: 2021-09-22 12:28:58 | |
| snapshots: | |
| 13321 .mirror.primary.c69f1f04-e722-4908-997e-e28dab64c59b.c361691e-9874-4b61-9d60-797e7e64c2f8 (peer_uuids:[3ab0dc51-3bbf-4440-8b19-0d9fe64818c2]) | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+replaying | |
| description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":15728640.0,"local_snapshot_timestamp":1632313598,"remote_snapshot_timestamp":1632313598,"replay_state":"idle"} | |
| service: a on west | |
| last_update: 2021-09-22 12:29:13 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+stopped | |
| description: local image is primary | |
| last_update: 2021-09-22 12:29:10 | |
| === Perform IO EAST === | |
| bench type write io_size 4096 io_threads 2 bytes 15728640 pattern sequential | |
| SEC OPS OPS/SEC BYTES/SEC | |
| elapsed: 3 ops: 3840 ops/sec: 1126.1 bytes/sec: 4.4 MiB/s | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Mirror image disable EAST === | |
| Mirroring disabled | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Image delete EAST === | |
| Removing image: 100% complete...done. | |
| === Ls pool EAST === | |
| === Ls pool WEST === | |
| test-1 | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+error | |
| description: split-brain | |
| service: a on west | |
| last_update: 2021-09-22 12:32:11 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+unknown | |
| description: remote image is non-primary | |
| last_update: 2021-09-22 12:34:40 | |
| === Sleeping for 150 seconds for snap schedule to catch up === | |
| === Ls pool WEST === | |
| test-1 | |
| === Mirror image status WEST === | |
| test-1: | |
| global_id: c69f1f04-e722-4908-997e-e28dab64c59b | |
| state: up+error | |
| description: split-brain | |
| service: a on west | |
| last_update: 2021-09-22 12:32:11 | |
| peer_sites: | |
| name: fa817e3a-0497-4e9a-b0f1-a77acfcbcac8 | |
| state: up+unknown | |
| description: remote image is non-primary | |
| last_update: 2021-09-22 12:37:10 | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment