Skip to content

Instantly share code, notes, and snippets.

@ideepika
Created December 23, 2021 10:32
Show Gist options
  • Save ideepika/0af014b8fcf9c131f8052268850468d8 to your computer and use it in GitHub Desktop.
Save ideepika/0af014b8fcf9c131f8052268850468d8 to your computer and use it in GitHub Desktop.
pwl_lock_failure.md
```
-4112> 2021-12-23T10:03:47.913+0000 7f7b09ffb700 1 lockdep using id 4092
-4111> 2021-12-23T10:03:47.913+0000 7f7b09ffb700 1 lockdep using id 4093
-4110> 2021-12-23T10:03:47.913+0000 7f7b09ffb700 20 librbd::io::Dispatcher: 0x5565b2d011c0 send: dispatch_spec=0x5565b2d8b5b0
-4109> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::io::QosImageDispatch: 0x5565b2d547f0 write: tid=11706, image_extents=[604999680,4096]
-4108> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::io::FlushTracker: 0x5565b2bdf7f0 start_io: tid=11706, flush_tid=3994
-4107> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::exclusive_lock::ImageDispatch: 0x7f7ad0004fb0 write: tid=11706, image_extents=[604999680,4096]
-4106> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::io::RefreshImageDispatch: 0x5565b2d537a0 write: tid=11706, image_extents=[604999680,4096]
-4105> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::io::WriteBlockImageDispatch: 0x5565b2be61c0 write: tid=11706
-4104> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::io::AioCompletion: 0x5565b2d7e1a0 set_request_count: pending=1
-4103> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::io::AsyncOperation: 0x5565b2d7e2b0 start_op
-4102> 2021-12-23T10:03:47.914+0000 7f7b09ffb700 20 librbd::BlockGuard: 0x7f7ad0008968 detain: [block_start = 604999680, block_end = 605003776), free_slots=2
-4101> 2021-12-23T10:03:47.955+0000 7f7b0a7fc700 1 lockdep using id 4094
-4100> 2021-12-23T10:03:47.955+0000 7f7b0a7fc700 1 lockdep using id 4095
-4099> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 failing miserably...
-4098> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 ERROR OUT OF IDS .. have 0 max 4096
-4097> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 0 log_obs
-4096> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 1 call::gate::lock
-4095> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 2 lock_dep_obs
-4094> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 3 mem_pool_obs
-4093> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 4 CrushLocation
-4092> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 5 ceph::io_context_pool::m
-4091> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 6 AuthRegistry::lock
-4090> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 7 MonClient::monc_lock
-4089> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 8 MgrClient::lock
-4088> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 9 librados::RadosClient::lock
-4087> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 10 PerfCounters::cct
-4086> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 11 PerfCounters::mempool
-4085> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 12 SimplePolicyMessenger::policy_lock
-4084> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 13 Messenger::DispatchQueue::lockradosclient
-4083> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 14 PerfCountersCollection
-4082> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 15 Messenger::DispatchQueue::local_delivery_lockradosclient
-4081> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 16 PerfCounters::throttle-msgr_dispatch_throttler-radosclient
-4080> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 17 PerfCounters::AsyncMessenger::Worker-0
-4079> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 18 PerfCounters::AsyncMessenger::Worker-1
-4078> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 19 PerfCounters::AsyncMessenger::Worker-2
-4077> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 20 AsyncMessenger::lock
-4076> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 21 CephContextServiceThread::_lock
-4075> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 22 AsyncMessenger::deleted_lock
-4074> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 23 Connection::lock
-4073> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 24 Objecter::pg_mapping_lock
-4072> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 25 Objecter::rwlock
-4071> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 26 PerfCounters::throttle-objecter_bytes
-4070> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 27 PerfCounters::throttle-objecter_ops
-4069> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 28 PerfCounters::objecter
-4068> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 29 RotatingKeyRing::lock
-4067> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 30 librados::IoCtxImpl::aio_write_list_lock
-4066> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 31 ConfigProxy::lock
-4065> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 32 librbd::ImageCtx::owner_lock (0x5565b2d0aef0)
-4064> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 33 librbd::ImageCtx::image_lock (0x5565b2d0aef0)
-4063> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 34 librbd::ImageCtx::timestamp_lock (0x5565b2d0aef0)
-4062> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 35 librbd::ImageCtx::async_ops_lock (0x5565b2d0aef0)
-4061> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 36 librbd::ImageCtx::copyup_list_lock (0x5565b2d0aef0)
-4060> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 37 Readahead::m_lock
-4059> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 38 Readahead::m_pending_lock
-4058> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 39 librbd::ImageState::m_lock (0x5565b2d526e0)
-4057> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 40 librbd::ImageUpdateWatchers::m_lock (0x5565b2d52960)
-4056> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 41 librbd::QuiesceWatchers::m_lock (0x5565b2bb3dc0)
-4055> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 42 librbd::Operations::m_queue_lock (0x7f7b00008480)
-4054> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 43 librbd::io::Dispatcher::lock (0x5565b2d011c0)
-4053> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 44 AsyncOpTracker::m_lock
-4052> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 45 librbd::io::FlushTracker::m_lock (0x5565b2be0040)
-4051> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 46 librbd::io::FlushTracker::m_lock (0x5565b2bdf7f0)
-4050> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 47 librbd::SafeTimerSingleton::lock
-4049> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 48 rbd_qos_iops_throttle_lock
-4048> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 49 rbd_qos_bps_throttle_lock
-4047> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 50 rbd_qos_read_iops_throttle_lock
-4046> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 51 rbd_qos_write_iops_throttle_lock
-4045> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 52 rbd_qos_read_bps_throttle_lock
-4044> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 53 rbd_qos_write_bps_throttle_lock
-4043> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 54 librbd::io::WriteBlockImageDispatch::m_lock (0x5565b2be61c0)
-4042> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 55 librbd::io::Dispatcher::lock (0x5565b2d56300)
-4041> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 56 SignalHandler::lock
-4040> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 57 PerfCounters::librbd-100d361bce14-test1-image1
-4039> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 58 librbd::Watcher::m_watch_lock (0x7f7ae4008780)
-4038> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 59 librbd::object_watcher::Notifier::m_aio_notify_lock (0x7f7ae4008840)
-4037> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 60 librbd::TaskFinisher::m_lock
-4036> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 61 Finisher::librbd::TaskFinisher::m_finisher
-4035> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 62 PerfCounters::finisher-librbd::TaskFinisher::m_finisher
-4034> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 63 librbd::ImageWatcher::m_async_request_lock (0x7f7ae4008780)
-4033> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 64 librbd::ImageWatcher::m_owner_client_id_lock (0x7f7ae4008780)
-4032> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 65 librbd::ManagedLock<I>::m_lock (0x7f7ad0001d48)
-4031> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 66 librbd::exclusve_lock::ImageDispatch::m_lock (0x7f7ad0004fb0)
-4030> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 67 librbd::io::FlushTracker::m_lock (0x7f7ad0002560)
-4029> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 68 PluginRegistery::lock
-4028> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 69 librbd::cache::WriteAroundObjectDispatch::lock (0x7f7ae4017140)
-4027> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 70 LingerOp::watch_lock #1
-4026> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 71 librbd::io::SimpleSchedulerObjectDispatch::lock (0x7f7ad0002460)
-4025> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 72 rbd_bencher::lock
-4024> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 73 librbd::ObjectMap::lock (0x7f7ad0002d70)
-4023> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 74 librbd::BlockGuard::m_lock
-4022> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 75 C_GatherBase::lock
-4021> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 76 librbd::cache::pwl::AbstractWriteLog::m_flush_guard_lock (0x7f7ad0008940)
-4020> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 77 librbd::cache::pwl::AbstractWriteLog::m_deferred_dispatch_lock (0x7f7ad0008940)
-4019> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 78 librbd::cache::pwl::AbstractWriteLog::m_blockguard_lock (0x7f7ad0008940)
-4018> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 79 librbd::cache::pwl::AbstractWriteLog::thread_pool::lock
-4017> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 80 librbd::cache::pwl::AbstractWriteLog::m_log_retire_lock (0x7f7ad0008940)
-4016> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 81 librbd::cache::pwl::AbstractWriteLog::m_entry_reader_lock
-4015> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 82 librbd::cache::pwl::AbstractWriteLog::m_log_append_lock (0x7f7ad0008940)
-4014> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 83 librbd::cache::pwl::AbstractWriteLog::m_lock (0x7f7ad0008940)
-4013> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 84 librbd::cache::pwl::LogMap::m_lock (0x7f7ad0009208)
-4012> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 85 ContextWQ::m_lock
-4011> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 86 PerfCounters::librbd-pwl-100d361bce14-test1-image1
-4010> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 87 HeartbeatMap::m_rwlock
-4009> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 88 librbd::cache::pwl::WriteLogEntry::m_entry_bl_lock (0x5565b2be7e80)
-4008> 2021-12-23T10:03:47.989+0000 7f7b09ffb700 0 lock 89 librbd::cache::pwl::WriteLogEntry::m_entry_bl_lock (0x7f7ae40106e0)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment