BUG #56147.md

Tracker: https://tracker.ceph.com/issues/56147

Suite: https://github.com/Matan-B/ceph/tree/octopus-only-update/qa/suites/upgrade/nautilus-x/parallel

Reproducing The Issue:

Teuthology run: https://pulpito.ceph.com/matan-2022-07-11_07:33:37-upgrade:nautilus-x:parallel-master-distro-basic-smithi/

1. Creating a Snapshot in Nautilus:

   POOL_NAME: pl2

   2022-07-11T07:50:12.779 INFO:teuthology.orchestra.run.smithi018.stdout:POOL_NAME         USED OBJECTS CLONES COPIES                     MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS  RD WR_OPS     WR USED COMPR UNDER COMPR
   2022-07-11T07:50:12.780 INFO:teuthology.orchestra.run.smithi018.stdout:pl2             16 KiB       2      1      4                     0       0        0      0 0 B      2  2 KiB        0 B         0 B
   

2. Upgrading to Octopus.

3. Creating a Snapshot in Octopus:

   POOL_NAME: oc_pool

   2022-07-11T07:58:04.655 INFO:teuthology.orchestra.run.smithi018.stdout:POOL_NAME                     USED  OBJECTS  CLONES  COPIES      MISSING_ON_PRIMARY  UNFOUND  DEGRADED  RD_OPS       RD  WR_OPS      WR  USED COMPR  UNDER COMPR
   2022-07-11T07:58:04.659 INFO:teuthology.orchestra.run.smithi018.stdout:oc_pool                     16 KiB        2       1       4                      0        0         0       0      0 B       2   2 KiB         0 B          0 B
   2022-07-11T07:58:04.659 INFO:teuthology.orchestra.run.smithi018.stdout:pl2                         16 KiB        2       1       4                      0        0         0       0      0 B       2   2 KiB         0 B          0 B
   

4. Removing both snapshots:

   2022-07-11T07:59:06.232 INFO:teuthology.orchestra.run.smithi018.stdout:POOL_NAME                     USED  OBJECTS  CLONES  COPIES      MISSING_ON_PRIMARY  UNFOUND  DEGRADED  RD_OPS       RD  WR_OPS      WR  USED COMPR  UNDER COMPR
   2022-07-11T07:59:06.235 INFO:teuthology.orchestra.run.smithi018.stdout:oc_pool                      8 KiB        1       0       2                      0        0         0       1    1 KiB       2   2 KiB         0 B          0 B
   2022-07-11T07:59:06.235 INFO:teuthology.orchestra.run.smithi018.stdout:pl2                         16 KiB        2       1       4                      0        0         0       1    1 KiB       2   2 KiB         0 B          0 B
   

   oc_pool CLONE objects: 0

   pl2 CLONE objects: 1

Notes:

• As shown, we are not able to remove the clone object. However, we can still read from it. This behaviour may occur since the object is still cached in SharedLRU<hobject_t, ObjectContext> object_contexts;. (Not verified yet) When setting osd_pg_object_context_cache_count to 0 we are still able to read from the clone object. Removing snapshots requires a valid SnapMapper. Disparate from reading which is done via the ONode interface, therfore it is not releated.
• The test sleeps for a few seconds before rados df to let the stats time to update.

Fixing the conversion:

PR: ceph/ceph#46908

Teuthology run: https://pulpito.ceph.com/matan-2022-07-11_08:25:26-upgrade:nautilus-x:parallel-master-distro-basic-smithi/

Same (1,2,3,4) steps result in:

2022-07-11T08:49:26.230 INFO:teuthology.orchestra.run.smithi055.stdout:oc_pool                      8 KiB        1       0       2                   0        0         0       1    1 KiB       2   2 KiB         0 B          0 B
2022-07-11T08:49:26.231 INFO:teuthology.orchestra.run.smithi055.stdout:pl2                          8 KiB        1       0       2                   0        0         0       1    1 KiB       2   2 KiB         0 B          0 B

oc_pool CLONE objects: 0

pl2 CLONE objects: 0 (!)

Converstion log:

2022-07-11T08:45:44.720+0000 7ff02b910d80 20 snap_mapper.convert_legacy convert MAP_0000000000000001_0000000000000004.ECF34CE6.1.objectone.. to SNA_4_0000000000000001_0000000000000004.ECF34CE6.1.objectone..