Skip to content

Instantly share code, notes, and snippets.

@NightDog

NightDog/gist:fd9b4b7b3e0c0c2ba29ce5d325bb97c6

Created June 13, 2021 15:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save NightDog/fd9b4b7b3e0c0c2ba29ce5d325bb97c6 to your computer and use it in GitHub Desktop.
Save NightDog/fd9b4b7b3e0c0c2ba29ce5d325bb97c6 to your computer and use it in GitHub Desktop.
Download ZIP
End of last successful run as well as first failed attempt
Raw
gistfile1.txt
Jun 10 14:49:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:49:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:49:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:49:57 ceph-node201 bash[16959]: L0 1/0 16.09 MB 0.2 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 215.91 184.03 1817 0.119 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 14:49:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 14:49:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: Sum 65/0 3.98 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.5 216.0 1133.71 853.22 2632 0.431 1256M 184M
Jun 10 14:49:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:49:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:49:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:49:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 14:49:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.86 184.03 1816 0.119 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: Uptime(secs): 7416604.9 total, 600.0 interval
Jun 10 14:49:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.395, interval 0.000
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 14:49:57 ceph-node201 bash[16959]: Cumulative compaction: 239.20 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.7 seconds
Jun 10 14:49:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 14:49:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 14:49:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 14:49:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:49:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:49:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:49:57 ceph-node201 bash[16959]: L0 1/0 16.09 MB 0.2 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 215.91 184.03 1817 0.119 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 14:49:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 14:49:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: Sum 65/0 3.98 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.5 216.0 1133.71 853.22 2632 0.431 1256M 184M
Jun 10 14:49:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:49:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:49:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:49:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 14:49:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.86 184.03 1816 0.119 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 14:49:57 ceph-node201 bash[16959]: Uptime(secs): 7416604.9 total, 0.0 interval
Jun 10 14:49:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.395, interval 0.000
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 14:49:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 14:49:57 ceph-node201 bash[16959]: Cumulative compaction: 239.20 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.7 seconds
Jun 10 14:49:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 14:49:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 14:49:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 14:59:57 ceph-node201 bash[16959]: debug 2021-06-10T14:59:57.288+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
Jun 10 14:59:57 ceph-node201 bash[16959]: debug 2021-06-10T14:59:57.288+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:778]
Jun 10 14:59:57 ceph-node201 bash[16959]: ** DB Stats **
Jun 10 14:59:57 ceph-node201 bash[16959]: Uptime(secs): 7417204.9 total, 600.0 interval
Jun 10 14:59:57 ceph-node201 bash[16959]: Cumulative writes: 119M writes, 480M keys, 119M commit groups, 1.0 writes per commit group, ingest: 427.79 GB, 0.06 MB/s
Jun 10 14:59:57 ceph-node201 bash[16959]: Cumulative WAL: 119M writes, 57M syncs, 2.09 writes per sync, written: 427.79 GB, 0.06 MB/s
Jun 10 14:59:57 ceph-node201 bash[16959]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 14:59:57 ceph-node201 bash[16959]: Interval writes: 2394 writes, 6054 keys, 2394 commit groups, 1.0 writes per commit group, ingest: 3.49 MB, 0.01 MB/s
Jun 10 14:59:57 ceph-node201 bash[16959]: Interval WAL: 2394 writes, 702 syncs, 3.41 writes per sync, written: 0.00 MB, 0.01 MB/s
Jun 10 14:59:57 ceph-node201 bash[16959]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 14:59:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:59:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:59:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:59:57 ceph-node201 bash[16959]: L0 1/0 16.09 MB 0.2 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 215.91 184.03 1817 0.119 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 14:59:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 14:59:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: Sum 65/0 3.98 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.5 216.0 1133.71 853.22 2632 0.431 1256M 184M
Jun 10 14:59:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:59:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:59:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:59:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 14:59:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.86 184.03 1816 0.119 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: Uptime(secs): 7417204.9 total, 600.0 interval
Jun 10 14:59:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.395, interval 0.000
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 14:59:57 ceph-node201 bash[16959]: Cumulative compaction: 239.20 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.7 seconds
Jun 10 14:59:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 14:59:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 14:59:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 14:59:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:59:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:59:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:59:57 ceph-node201 bash[16959]: L0 1/0 16.09 MB 0.2 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 215.91 184.03 1817 0.119 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 14:59:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 14:59:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: Sum 65/0 3.98 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.5 216.0 1133.71 853.22 2632 0.431 1256M 184M
Jun 10 14:59:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 14:59:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 14:59:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 14:59:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 14:59:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.86 184.03 1816 0.119 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 14:59:57 ceph-node201 bash[16959]: Uptime(secs): 7417204.9 total, 0.0 interval
Jun 10 14:59:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.395, interval 0.000
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 14:59:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 14:59:57 ceph-node201 bash[16959]: Cumulative compaction: 239.20 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.7 seconds
Jun 10 14:59:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 14:59:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 14:59:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.474+0000 7f9a115e7700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 7553 from recycle list
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.474+0000 7f9a115e7700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #7565. Immutable memtables: 0.
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.474+0000 7f9a17df4700 4 rocksdb: (Original Log Time 2021/06/10-15:06:53.478155) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1,
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.474+0000 7f9a17df4700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 2668] Flushing memtable with next log file: 7565
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.474+0000 7f9a17df4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1623337613478218, "job": 2668, "event": "flush_started", "num_memtables": 1, "num_entries": 394401, "num_deletes": 98337, "total_data_size": 248981330, "memory_usage": 2600542
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.474+0000 7f9a17df4700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 2668] Level-0 flush table #7566: started
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.594+0000 7f9a17df4700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1623337613599939, "cf_name": "default", "job": 2668, "event": "table_file_creation", "file_number": 7566, "file_size": 16334776, "table_properties": {"data_size": 15949819, "i
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.594+0000 7f9a17df4700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 2668] Level-0 flush table #7566: 16334776 bytes OK
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.598+0000 7f9a17df4700 4 rocksdb: [db/db_impl_files.cc:194] adding log 7563 to recycle list
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.598+0000 7f9a17df4700 4 rocksdb: (Original Log Time 2021/06/10-15:06:53.599993) [db/memtable_list.cc:380] [default] Level-0 commit table #7566 started
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.598+0000 7f9a17df4700 4 rocksdb: (Original Log Time 2021/06/10-15:06:53.600401) [db/memtable_list.cc:428] [default] Level-0 commit table #7566: memtable #1 done
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.598+0000 7f9a17df4700 4 rocksdb: (Original Log Time 2021/06/10-15:06:53.600439) EVENT_LOG_v1 {"time_micros": 1623337613600409, "job": 2668, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 4, 40, 20, 0,
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.598+0000 7f9a17df4700 4 rocksdb: (Original Log Time 2021/06/10-15:06:53.600471) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[2 4 40 20 0 0 0] max score 1.00
Jun 10 15:06:53 ceph-node201 bash[16959]: debug 2021-06-10T15:06:53.598+0000 7f9a17df4700 4 rocksdb: [db/db_impl_files.cc:356] [JOB 2668] Try to delete WAL files size 247999839, prev total WAL file size 248005201, number of live WAL files 2.
Jun 10 15:09:57 ceph-node201 bash[16959]: debug 2021-06-10T15:09:57.289+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
Jun 10 15:09:57 ceph-node201 bash[16959]: debug 2021-06-10T15:09:57.289+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:778]
Jun 10 15:09:57 ceph-node201 bash[16959]: ** DB Stats **
Jun 10 15:09:57 ceph-node201 bash[16959]: Uptime(secs): 7417804.9 total, 600.0 interval
Jun 10 15:09:57 ceph-node201 bash[16959]: Cumulative writes: 119M writes, 480M keys, 119M commit groups, 1.0 writes per commit group, ingest: 427.79 GB, 0.06 MB/s
Jun 10 15:09:57 ceph-node201 bash[16959]: Cumulative WAL: 119M writes, 57M syncs, 2.09 writes per sync, written: 427.79 GB, 0.06 MB/s
Jun 10 15:09:57 ceph-node201 bash[16959]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 15:09:57 ceph-node201 bash[16959]: Interval writes: 2164 writes, 5274 keys, 2164 commit groups, 1.0 writes per commit group, ingest: 3.27 MB, 0.01 MB/s
Jun 10 15:09:57 ceph-node201 bash[16959]: Interval WAL: 2164 writes, 721 syncs, 3.00 writes per sync, written: 0.00 MB, 0.01 MB/s
Jun 10 15:09:57 ceph-node201 bash[16959]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 15:09:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:09:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:09:57 ceph-node201 bash[16959]: L0 2/0 31.67 MB 0.5 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 216.03 184.14 1818 0.119 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 15:09:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:09:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.9 0.12 0.11 1 0.122 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:09:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:09:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Uptime(secs): 7417804.9 total, 600.0 interval
Jun 10 15:09:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.015
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Interval compaction: 0.02 GB write, 0.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:09:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:09:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:09:57 ceph-node201 bash[16959]: L0 2/0 31.67 MB 0.5 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 216.03 184.14 1818 0.119 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 15:09:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:09:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:09:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:09:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Uptime(secs): 7417804.9 total, 0.0 interval
Jun 10 15:09:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.000
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:09:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: debug 2021-06-10T15:19:57.290+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
Jun 10 15:19:57 ceph-node201 bash[16959]: debug 2021-06-10T15:19:57.290+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:778]
Jun 10 15:19:57 ceph-node201 bash[16959]: ** DB Stats **
Jun 10 15:19:57 ceph-node201 bash[16959]: Uptime(secs): 7418404.9 total, 600.0 interval
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative writes: 119M writes, 480M keys, 119M commit groups, 1.0 writes per commit group, ingest: 427.80 GB, 0.06 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative WAL: 119M writes, 57M syncs, 2.09 writes per sync, written: 427.80 GB, 0.06 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval writes: 2124 writes, 5196 keys, 2124 commit groups, 1.0 writes per commit group, ingest: 3.17 MB, 0.01 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval WAL: 2124 writes, 704 syncs, 3.01 writes per sync, written: 0.00 MB, 0.01 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: L0 2/0 31.67 MB 0.5 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 216.03 184.14 1818 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 15:19:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:19:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Uptime(secs): 7418404.9 total, 600.0 interval
Jun 10 15:19:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:19:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: L0 2/0 31.67 MB 0.5 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 216.03 184.14 1818 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 15:19:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:19:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Uptime(secs): 7418404.9 total, 0.0 interval
Jun 10 15:19:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:19:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:29:13 ceph-node201 bash[16959]: debug 2021-06-10T15:29:13.980+0000 7f9a20605700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
Jun 10 15:29:13 ceph-node201 bash[16959]: debug 2021-06-10T15:29:13.980+0000 7f9a20605700 -1 osd.83 8161029 *** Got signal Terminated ***
Jun 10 15:29:13 ceph-node201 bash[16959]: debug 2021-06-10T15:29:13.980+0000 7f9a20605700 -1 osd.83 8161029 *** Immediate shutdown (osd_fast_shutdown=true) ***
Jun 10 15:29:15 ceph-node201 docker[31292]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 15:29:17 ceph-node201 docker[31292]: Error response from daemon: No such container: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf-osd.83
Jun 10 15:29:17 ceph-node201 bash[31574]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 15:29:17 ceph-node201 bash[31574]: Error: No such container: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf-osd.83-deactivate
Jun 10 15:29:18 ceph-node201 bash[31574]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 18:07:13 ceph-node201 systemd[1]: Started Ceph osd.83 for f4495398-a8c4-4ad9-8219-80c48625abdf.
Jun 10 18:07:13 ceph-node201 bash[26687]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 18:07:13 ceph-node201 bash[26687]: Error: No such container: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf-osd.83-activate
Jun 10 15:09:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:09:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.9 0.12 0.11 1 0.122 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:09:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:09:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Uptime(secs): 7417804.9 total, 600.0 interval
Jun 10 15:09:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.015
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Interval compaction: 0.02 GB write, 0.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:09:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:09:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:09:57 ceph-node201 bash[16959]: L0 2/0 31.67 MB 0.5 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 216.03 184.14 1818 0.119 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 15:09:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:09:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:09:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:09:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:09:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:09:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Uptime(secs): 7417804.9 total, 0.0 interval
Jun 10 15:09:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.000
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:09:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 15:09:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:09:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: debug 2021-06-10T15:19:57.290+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
Jun 10 15:19:57 ceph-node201 bash[16959]: debug 2021-06-10T15:19:57.290+0000 7f9a0fde4700 4 rocksdb: [db/db_impl.cc:778]
Jun 10 15:19:57 ceph-node201 bash[16959]: ** DB Stats **
Jun 10 15:19:57 ceph-node201 bash[16959]: Uptime(secs): 7418404.9 total, 600.0 interval
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative writes: 119M writes, 480M keys, 119M commit groups, 1.0 writes per commit group, ingest: 427.80 GB, 0.06 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative WAL: 119M writes, 57M syncs, 2.09 writes per sync, written: 427.80 GB, 0.06 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval writes: 2124 writes, 5196 keys, 2124 commit groups, 1.0 writes per commit group, ingest: 3.17 MB, 0.01 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval WAL: 2124 writes, 704 syncs, 3.01 writes per sync, written: 0.00 MB, 0.01 MB/s
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: L0 2/0 31.67 MB 0.5 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 216.03 184.14 1818 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 15:19:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:19:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Uptime(secs): 7418404.9 total, 600.0 interval
Jun 10 15:19:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:19:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: L0 2/0 31.67 MB 0.5 0.0 0.0 0.0 28.4 28.4 0.0 1.0 0.0 134.7 216.03 184.14 1818 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: L1 4/0 227.52 MB 0.9 85.9 28.4 57.5 75.4 17.9 0.0 2.7 226.8 199.1 387.75 294.64 457 0.848 525M 44M
Jun 10 15:19:57 ceph-node201 bash[16959]: L2 40/0 2.49 GB 1.0 149.3 16.9 132.5 135.4 3.0 0.8 8.0 288.5 261.6 530.05 374.55 358 1.481 730M 139M
Jun 10 15:19:57 ceph-node201 bash[16959]: L3 20/0 1.25 GB 0.1 0.0 0.0 0.0 0.0 0.0 1.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Sum 66/0 4.00 GB 0.0 235.2 45.2 190.0 239.2 49.2 2.0 8.4 212.4 216.0 1133.83 853.32 2633 0.431 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: ** Compaction Stats [default] **
Jun 10 15:19:57 ceph-node201 bash[16959]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
Jun 10 15:19:57 ceph-node201 bash[16959]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Jun 10 15:19:57 ceph-node201 bash[16959]: Low 0/0 0.00 KB 0.0 235.2 45.2 190.0 210.8 20.8 0.0 0.0 262.4 235.2 917.80 669.18 815 1.126 1256M 184M
Jun 10 15:19:57 ceph-node201 bash[16959]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 28.4 28.4 0.0 0.0 0.0 134.7 215.98 184.14 1817 0.119 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 198.0 0.05 0.00 1 0.048 0 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Uptime(secs): 7418404.9 total, 0.0 interval
Jun 10 15:19:57 ceph-node201 bash[16959]: Flush(GB): cumulative 28.410, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(GB): cumulative 0.000, interval 0.000
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Total Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(L0 Files): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: AddFile(Keys): cumulative 0, interval 0
Jun 10 15:19:57 ceph-node201 bash[16959]: Cumulative compaction: 239.21 GB write, 0.03 MB/s write, 235.21 GB read, 0.03 MB/s read, 1133.8 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Jun 10 15:19:57 ceph-node201 bash[16959]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total co
Jun 10 15:19:57 ceph-node201 bash[16959]: ** File Read Latency Histogram By Level [default] **
Jun 10 15:29:13 ceph-node201 bash[16959]: debug 2021-06-10T15:29:13.980+0000 7f9a20605700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
Jun 10 15:29:13 ceph-node201 bash[16959]: debug 2021-06-10T15:29:13.980+0000 7f9a20605700 -1 osd.83 8161029 *** Got signal Terminated ***
Jun 10 15:29:13 ceph-node201 bash[16959]: debug 2021-06-10T15:29:13.980+0000 7f9a20605700 -1 osd.83 8161029 *** Immediate shutdown (osd_fast_shutdown=true) ***
Jun 10 15:29:15 ceph-node201 docker[31292]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 15:29:17 ceph-node201 docker[31292]: Error response from daemon: No such container: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf-osd.83
Jun 10 15:29:17 ceph-node201 bash[31574]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 15:29:17 ceph-node201 bash[31574]: Error: No such container: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf-osd.83-deactivate
Jun 10 15:29:18 ceph-node201 bash[31574]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 18:07:13 ceph-node201 systemd[1]: Started Ceph osd.83 for f4495398-a8c4-4ad9-8219-80c48625abdf.
Jun 10 18:07:13 ceph-node201 bash[26687]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 18:07:13 ceph-node201 bash[26687]: Error: No such container: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf-osd.83-activate
Jun 10 18:07:14 ceph-node201 bash[26687]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-83
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-ff60b68a-26fe-4294-8bec-4a9c329e858d/osd-block-73ab12e6-7758-4ebe-9319-5935309fcacd --path /var/lib/ceph/osd/ceph-83 --no-mon-config
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/ln -snf /dev/ceph-ff60b68a-26fe-4294-8bec-4a9c329e858d/osd-block-73ab12e6-7758-4ebe-9319-5935309fcacd /var/lib/ceph/osd/ceph-83/block
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-83/block
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--ff60b68a--26fe--4294--8bec--4a9c329e858d-osd--block--73ab12e6--7758--4ebe--9319--5935309fcacd
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-83
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/ln -snf /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be /var/lib/ceph/osd/ceph-83/block.db
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-00ed472c-f900-4dc3-9ddc-0e2f3b6547e3/osd-db-bb0eaa16-a1e0-4985-b4bd-74799e5226be
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--00ed472c--f900--4dc3--9ddc--0e2f3b6547e3-osd--db--bb0eaa16--a1e0--4985--b4bd--74799e5226be
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-83/block.db
Jun 10 18:07:15 ceph-node201 bash[26687]: Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--00ed472c--f900--4dc3--9ddc--0e2f3b6547e3-osd--db--bb0eaa16--a1e0--4985--b4bd--74799e5226be
Jun 10 18:07:15 ceph-node201 bash[26687]: --> ceph-volume lvm activate successful for osd ID: 83
Jun 10 18:07:16 ceph-node201 bash[26687]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 18:07:16 ceph-node201 bash[26687]: Error: No such container: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf-osd.83
Jun 10 18:07:16 ceph-node201 bash[26687]: WARNING: Error loading config file: .dockercfg: $HOME is not defined
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.193+0000 7f7be492af40 0 set uid:gid to 167:167 (ceph:ceph)
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.193+0000 7f7be492af40 0 ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process ceph-osd, pid 1
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.193+0000 7f7be492af40 0 pidfile_write: ignore empty --pid-file
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.193+0000 7f7be492af40 1 bdev create path /var/lib/ceph/osd/ceph-83/block type kernel
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.193+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) open path /var/lib/ceph/osd/ceph-83/block
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 1 bluestore(/var/lib/ceph/osd/ceph-83) _set_cache_sizes cache_size 536870912 meta 0.4 kv 0.4 data 0.2
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 1 bdev create path /var/lib/ceph/osd/ceph-83/block.db type kernel
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) open path /var/lib/ceph/osd/ceph-83/block.db
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) open size 85899345920 (0x1400000000, 80 GiB) block_size 4096 (4 KiB) non-rotational discard supported
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-83/block.db size 80 GiB
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 -1 bluestore(/var/lib/ceph/osd/ceph-83) _minimal_open_bluefs check block device(/var/lib/ceph/osd/ceph-83/block.db) label returned: (2) No such file or directory
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.197+0000 7f7be492af40 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) close
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.477+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) close
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.729+0000 7f7be492af40 0 starting osd.83 osd_data /var/lib/ceph/osd/ceph-83 /var/lib/ceph/osd/ceph-83/journal
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.729+0000 7f7be492af40 -1 Falling back to public interface
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.737+0000 7f7be492af40 0 load: jerasure load: lrc load: isa
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.737+0000 7f7be492af40 1 bdev create path /var/lib/ceph/osd/ceph-83/block type kernel
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.737+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) open path /var/lib/ceph/osd/ceph-83/block
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.737+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.737+0000 7f7be492af40 1 bluestore(/var/lib/ceph/osd/ceph-83) _set_cache_sizes cache_size 536870912 meta 0.4 kv 0.4 data 0.2
Jun 10 18:07:17 ceph-node201 bash[26687]: debug 2021-06-10T18:07:17.737+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) close
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 0 osd.83:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 0 osd.83:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 0 osd.83:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 0 osd.83:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 0 osd.83:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bluestore(/var/lib/ceph/osd/ceph-83) _mount path /var/lib/ceph/osd/ceph-83
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bdev create path /var/lib/ceph/osd/ceph-83/block type kernel
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) open path /var/lib/ceph/osd/ceph-83/block
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) open size 4000783007744 (0x3a381400000, 3.6 TiB) block_size 4096 (4 KiB) rotational discard not supported
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bluestore(/var/lib/ceph/osd/ceph-83) _set_cache_sizes cache_size 536870912 meta 0.4 kv 0.4 data 0.2
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bdev create path /var/lib/ceph/osd/ceph-83/block.db type kernel
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) open path /var/lib/ceph/osd/ceph-83/block.db
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) open size 85899345920 (0x1400000000, 80 GiB) block_size 4096 (4 KiB) non-rotational discard supported
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-83/block.db size 80 GiB
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 -1 bluestore(/var/lib/ceph/osd/ceph-83) _minimal_open_bluefs check block device(/var/lib/ceph/osd/ceph-83/block.db) label returned: (2) No such file or directory
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.029+0000 7f7be492af40 1 bdev(0x563d8822a700 /var/lib/ceph/osd/ceph-83/block.db) close
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.313+0000 7f7be492af40 1 bdev(0x563d8822a000 /var/lib/ceph/osd/ceph-83/block) close
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.569+0000 7f7be492af40 -1 osd.83 0 OSD:init: unable to mount object store
Jun 10 18:07:18 ceph-node201 bash[26687]: debug 2021-06-10T18:07:18.569+0000 7f7be492af40 -1 ** ERROR: osd init failed: (2) No such file or directory
Jun 10 18:07:19 ceph-node201 systemd[1]: ceph-f4495398-a8c4-4ad9-8219-80c48625abdf@osd.83.service: Main process exited, code=exited, status=1/FAILURE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment