Skip to content

Instantly share code, notes, and snippets.

@filimonov
Created September 16, 2021 15:33
Show Gist options
  • Save filimonov/85b894268f978c2ccc18ea69bae5adbd to your computer and use it in GitHub Desktop.
Save filimonov/85b894268f978c2ccc18ea69bae5adbd to your computer and use it in GitHub Desktop.
[51140.383890] perf: interrupt took too long (2519 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
[59014.200296] perf: interrupt took too long (3150 > 3148), lowering kernel.perf_event_max_sample_rate to 63000
[59445.940213] perf: interrupt took too long (3939 > 3937), lowering kernel.perf_event_max_sample_rate to 50000
[79341.838396] XFS (dm-1): Mounting V5 Filesystem
[79341.862314] XFS (dm-1): Ending clean mount
[79342.489941] XFS (dm-0): Mounting V5 Filesystem
[79342.510930] XFS (dm-0): Ending clean mount
[79343.419111] XFS (dm-0): Unmounting Filesystem
[79343.451992] XFS (dm-1): Unmounting Filesystem
[85991.320353] XFS (dm-1): Mounting V5 Filesystem
[85991.342594] XFS (dm-1): Ending clean mount
[85991.982472] XFS (dm-0): Mounting V5 Filesystem
[85992.004335] XFS (dm-0): Ending clean mount
[85992.907717] XFS (dm-0): Unmounting Filesystem
[85992.935703] XFS (dm-1): Unmounting Filesystem
[102694.188000] perf: interrupt took too long (4941 > 4923), lowering kernel.perf_event_max_sample_rate to 40000
[126340.883673] CPU57: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883675] CPU49: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883676] CPU51: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883677] CPU15: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883678] CPU43: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883679] CPU7: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883680] CPU61: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883682] CPU37: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883683] CPU19: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883684] CPU55: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883685] CPU1: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883686] CPU63: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883687] CPU27: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883688] CPU21: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883689] CPU69: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883690] CPU33: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883691] CPU23: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883692] CPU59: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883693] CPU13: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883695] CPU41: Core temperature above threshold, cpu clock throttled (total events = 1)
[126340.883695] CPU5: Core temperature above threshold, cpu clock throttled (total events = 1)
[126340.883697] CPU53: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883697] CPU17: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883699] CPU67: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883700] CPU31: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883701] CPU39: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883702] CPU3: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883703] CPU71: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883704] CPU35: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883706] CPU29: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883706] CPU65: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883708] CPU45: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883708] CPU9: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883709] CPU25: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883710] CPU47: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883711] CPU11: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883712] CPU41: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.883712] CPU5: Package temperature above threshold, cpu clock throttled (total events = 1)
[126340.885707] CPU43: Package temperature/speed normal
[126340.885709] CPU47: Package temperature/speed normal
[126340.885709] CPU13: Package temperature/speed normal
[126340.885710] CPU11: Package temperature/speed normal
[126340.885711] CPU53: Package temperature/speed normal
[126340.885712] CPU71: Package temperature/speed normal
[126340.885712] CPU15: Package temperature/speed normal
[126340.885713] CPU21: Package temperature/speed normal
[126340.885714] CPU59: Package temperature/speed normal
[126340.885714] CPU23: Package temperature/speed normal
[126340.885715] CPU69: Package temperature/speed normal
[126340.885715] CPU33: Package temperature/speed normal
[126340.885716] CPU41: Core temperature/speed normal
[126340.885717] CPU5: Core temperature/speed normal
[126340.885717] CPU27: Package temperature/speed normal
[126340.885718] CPU63: Package temperature/speed normal
[126340.885719] CPU55: Package temperature/speed normal
[126340.885719] CPU31: Package temperature/speed normal
[126340.885720] CPU67: Package temperature/speed normal
[126340.885720] CPU65: Package temperature/speed normal
[126340.885721] CPU29: Package temperature/speed normal
[126340.885722] CPU49: Package temperature/speed normal
[126340.885722] CPU51: Package temperature/speed normal
[126340.885723] CPU3: Package temperature/speed normal
[126340.885724] CPU9: Package temperature/speed normal
[126340.885725] CPU35: Package temperature/speed normal
[126340.885725] CPU17: Package temperature/speed normal
[126340.885726] CPU19: Package temperature/speed normal
[126340.885727] CPU45: Package temperature/speed normal
[126340.885727] CPU25: Package temperature/speed normal
[126340.885728] CPU7: Package temperature/speed normal
[126340.885728] CPU37: Package temperature/speed normal
[126340.885729] CPU1: Package temperature/speed normal
[126340.885730] CPU39: Package temperature/speed normal
[126340.885730] CPU61: Package temperature/speed normal
[126340.885731] CPU41: Package temperature/speed normal
[126340.885731] CPU5: Package temperature/speed normal
[126341.271473] CPU57: Package temperature/speed normal
[127490.659951] perf: interrupt took too long (6184 > 6176), lowering kernel.perf_event_max_sample_rate to 32000
[129648.987293] INFO: task systemd:1 blocked for more than 120 seconds.
[129648.998027] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129649.010457] systemd D ffff9fa1adb78000 0 1 0 0x00000000
[129649.010466] Call Trace:
[129649.017564] [<ffffffff84d50ea5>] ? blk_mq_start_request+0x75/0x160
[129649.028625] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129649.038449] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0
[129649.049177] [<ffffffff8516c36d>] wait_for_completion+0xfd/0x140
[129649.060141] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20
[129649.070502] [<ffffffff84abb93d>] flush_work+0xfd/0x190
[129649.080321] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90
[129649.090989] [<ffffffffc07367ea>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[129649.102243] [<ffffffffc0734b90>] _xfs_log_force_lsn+0x80/0x340 [xfs]
[129649.113546] [<ffffffff84d48a64>] ? blk_finish_plug+0x14/0x40
[129649.124260] [<ffffffffc07229bc>] ? __xfs_iunpin_wait+0x9c/0x150 [xfs]
[129649.135702] [<ffffffffc0734e84>] xfs_log_force_lsn+0x34/0x70 [xfs]
[129649.146794] [<ffffffffc0725cf9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[129649.157933] [<ffffffffc07229bc>] __xfs_iunpin_wait+0x9c/0x150 [xfs]
[129649.169195] [<ffffffff84ac4230>] ? wake_bit_function+0x40/0x40
[129649.180125] [<ffffffffc0725cf9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[129649.191145] [<ffffffffc0719543>] xfs_reclaim_inode+0x143/0x360 [xfs]
[129649.202410] [<ffffffffc07199c7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[129649.214108] [<ffffffffc071aba3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[129649.225690] [<ffffffffc072b1f5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[129649.237822] [<ffffffff84c451d6>] prune_super+0xf6/0x190
[129649.248285] [<ffffffff84bcb375>] shrink_slab+0x175/0x340
[129649.258919] [<ffffffff84c385d1>] ? vmpressure+0x21/0x90
[129649.269536] [<ffffffff84bce632>] do_try_to_free_pages+0x3c2/0x4e0
[129649.281135] [<ffffffff84bce84c>] try_to_free_pages+0xfc/0x180
[129649.292273] [<ffffffff8516158c>] __alloc_pages_slowpath+0x457/0x724
[129649.303886] [<ffffffff84bc2744>] __alloc_pages_nodemask+0x404/0x420
[129649.315359] [<ffffffff84a960fd>] copy_process+0x1dd/0x1a40
[129649.326132] [<ffffffff84a97b11>] do_fork+0x91/0x320
[129649.336368] [<ffffffff84cfa62c>] ? security_file_fcntl+0x1c/0x20
[129649.347857] [<ffffffff84c5610f>] ? SyS_fcntl+0xef/0x6f0
[129649.358651] [<ffffffff84a97e26>] SyS_clone+0x16/0x20
[129649.369056] [<ffffffff851791b4>] stub_clone+0x44/0x70
[129649.379445] [<ffffffff85178ddb>] ? system_call_fastpath+0x22/0x27
[129649.391010] INFO: task kswapd0:390 blocked for more than 120 seconds.
[129649.402940] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129649.416485] kswapd0 D ffffa02e7e0bc100 0 390 2 0x00000000
[129649.416488] Call Trace:
[129649.424700] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129649.435408] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0
[129649.446883] [<ffffffff8516c36d>] wait_for_completion+0xfd/0x140
[129649.458629] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20
[129649.469992] [<ffffffff84abb93d>] flush_work+0xfd/0x190
[129649.481094] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90
[129649.492805] [<ffffffffc07367ea>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[129649.504940] [<ffffffffc0734b90>] _xfs_log_force_lsn+0x80/0x340 [xfs]
[129649.516844] [<ffffffff84d48a64>] ? blk_finish_plug+0x14/0x40
[129649.527987] [<ffffffffc07229bc>] ? __xfs_iunpin_wait+0x9c/0x150 [xfs]
[129649.539830] [<ffffffffc0734e84>] xfs_log_force_lsn+0x34/0x70 [xfs]
[129649.551384] [<ffffffffc0725cf9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[129649.562905] [<ffffffffc07229bc>] __xfs_iunpin_wait+0x9c/0x150 [xfs]
[129649.574477] [<ffffffff84ac4230>] ? wake_bit_function+0x40/0x40
[129649.585656] [<ffffffffc0725cf9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[129649.597041] [<ffffffffc0719543>] xfs_reclaim_inode+0x143/0x360 [xfs]
[129649.608792] [<ffffffffc07199c7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[129649.620884] [<ffffffffc071aba3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[129649.632751] [<ffffffffc072b1f5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[129649.645115] [<ffffffff84c451d6>] prune_super+0xf6/0x190
[129649.655698] [<ffffffff84bcb375>] shrink_slab+0x175/0x340
[129649.666333] [<ffffffff84c38637>] ? vmpressure+0x87/0x90
[129649.676836] [<ffffffff84bcf171>] balance_pgdat+0x4b1/0x5e0
[129649.687620] [<ffffffff84bcf413>] kswapd+0x173/0x440
[129649.697813] [<ffffffff84ac4170>] ? wake_up_atomic_t+0x30/0x30
[129649.708864] [<ffffffff84bcf2a0>] ? balance_pgdat+0x5e0/0x5e0
[129649.719810] [<ffffffff84ac30a1>] kthread+0xd1/0xe0
[129649.729852] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40
[129649.741153] [<ffffffff85178c1d>] ret_from_fork_nospec_begin+0x7/0x21
[129649.752870] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40
[129649.764249] INFO: task xfsaild/dm-6:40642 blocked for more than 120 seconds.
[129649.776615] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129649.789896] xfsaild/dm-6 D ffffa0be79534100 0 40642 2 0x00000000
[129649.789899] Call Trace:
[129649.798045] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90
[129649.809522] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129649.820026] [<ffffffffc0736942>] xlog_cil_force_lsn+0x1e2/0x210 [xfs]
[129649.832230] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20
[129649.843449] [<ffffffffc0734765>] _xfs_log_force+0x85/0x2c0 [xfs]
[129649.855161] [<ffffffff84aaad50>] ? __internal_add_timer+0x130/0x130
[129649.867195] [<ffffffffc0740aac>] ? xfsaild+0x16c/0x6f0 [xfs]
[129649.878681] [<ffffffffc07349cc>] xfs_log_force+0x2c/0x70 [xfs]
[129649.890392] [<ffffffffc0740940>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[129649.903541] [<ffffffffc0740aac>] xfsaild+0x16c/0x6f0 [xfs]
[129649.914886] [<ffffffffc0740940>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[129649.927945] [<ffffffff84ac30a1>] kthread+0xd1/0xe0
[129649.938584] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40
[129649.950506] [<ffffffff85178c1d>] ret_from_fork_nospec_begin+0x7/0x21
[129649.962586] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40
[129649.974330] INFO: task systemd-logind:40802 blocked for more than 120 seconds.
[129649.987306] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129650.001000] systemd-logind D ffffa02e68b68000 0 40802 1 0x00000000
[129650.001002] Call Trace:
[129650.009297] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129650.020182] [<ffffffff8516d8a5>] rwsem_down_write_failed+0x225/0x3a0
[129650.032595] [<ffffffff84d77fc2>] ? get_from_free_list+0x42/0x50
[129650.044561] [<ffffffff84d887a7>] call_rwsem_down_write_failed+0x17/0x30
[129650.057279] [<ffffffff84d79260>] ? ida_get_new_above+0x230/0x2a0
[129650.069372] [<ffffffff8516b2ed>] down_write+0x2d/0x3d
[129650.080479] [<ffffffff84bca831>] register_shrinker+0x21/0x50
[129650.092218] [<ffffffff84c46429>] sget_userns+0x489/0x4c0
[129650.103616] [<ffffffff84c45380>] ? get_anon_bdev+0x110/0x110
[129650.115387] [<ffffffff84c464dd>] sget+0x7d/0xb0
[129650.125974] [<ffffffff84c45380>] ? get_anon_bdev+0x110/0x110
[129650.137700] [<ffffffff84bd1330>] ? shmem_rename+0x20/0x20
[129650.149133] [<ffffffff84c46540>] mount_nodev+0x30/0xb0
[129650.160310] [<ffffffff84bd0fb8>] shmem_mount+0x18/0x20
[129650.171477] [<ffffffff84c470de>] mount_fs+0x3e/0x1b0
[129650.182440] [<ffffffff84c64d07>] vfs_kern_mount+0x67/0x110
[129650.193886] [<ffffffff84c6732f>] do_mount+0x1ef/0xce0
[129650.204895] [<ffffffff84c67065>] ? copy_mount_options+0xc5/0x170
[129650.216916] [<ffffffff84c68163>] SyS_mount+0x83/0xd0
[129650.227871] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27
[129650.239862] INFO: task console:60319 blocked for more than 120 seconds.
[129650.252502] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129650.266498] console D ffffa0be2fa3b0c0 0 60319 60311 0x20020000
[129650.266501] Call Trace:
[129650.275091] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90
[129650.287279] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129650.298476] [<ffffffffc0736942>] xlog_cil_force_lsn+0x1e2/0x210 [xfs]
[129650.311263] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20
[129650.323048] [<ffffffffc0734b90>] _xfs_log_force_lsn+0x80/0x340 [xfs]
[129650.335626] [<ffffffff84bb6f6f>] ? filemap_fdatawait_range+0x1f/0x30
[129650.348135] [<ffffffff8516b292>] ? down_read+0x12/0x40
[129650.359385] [<ffffffffc0714f5d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
[129650.371519] [<ffffffff84c77207>] do_fsync+0x67/0xb0
[129650.383675] [<ffffffff84c774f0>] SyS_fsync+0x10/0x20
[129650.394706] [<ffffffff8517cbb0>] sysenter_dispatch+0xd/0x2b
[129650.406469] INFO: task console:62065 blocked for more than 120 seconds.
[129650.419099] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129650.433357] console D ffffa02e62751040 0 62065 60311 0x20020000
[129650.433359] Call Trace:
[129650.441948] [<ffffffff8516ced9>] schedule_preempt_disabled+0x29/0x70
[129650.454598] [<ffffffff8516ae57>] __mutex_lock_slowpath+0xc7/0x1d0
[129650.466960] [<ffffffff8516a23f>] mutex_lock+0x1f/0x2f
[129650.478215] [<ffffffff84c44057>] compat_SyS_writev+0x67/0xf0
[129650.490048] [<ffffffff8517cbb0>] sysenter_dispatch+0xd/0x2b
[129650.501787] INFO: task console:62078 blocked for more than 120 seconds.
[129650.514554] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129650.528628] console D ffffa02e62752080 0 62078 60311 0x20020000
[129650.528630] Call Trace:
[129650.537212] [<ffffffff84c1da62>] ? kmem_cache_alloc+0x1c2/0x1f0
[129650.549406] [<ffffffffc0730f07>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[129650.561841] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129650.572865] [<ffffffff8516d8a5>] rwsem_down_write_failed+0x225/0x3a0
[129650.585424] [<ffffffffc07315c4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
[129650.598427] [<ffffffffc072057d>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
[129650.611177] [<ffffffff84d887a7>] call_rwsem_down_write_failed+0x17/0x30
[129650.624015] [<ffffffff8516b2ed>] down_write+0x2d/0x3d
[129650.635283] [<ffffffffc07232d4>] xfs_ilock+0xc4/0x120 [xfs]
[129650.647066] [<ffffffffc072057d>] xfs_vn_update_time+0xcd/0x150 [xfs]
[129650.659646] [<ffffffff84c5f4e8>] update_time+0x28/0xd0
[129650.671022] [<ffffffff84c5f630>] file_update_time+0xa0/0xf0
[129650.682826] [<ffffffffc071612d>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
[129650.696194] [<ffffffffc071675a>] xfs_file_buffered_aio_write+0xca/0x2c0 [xfs]
[129650.709678] [<ffffffffc0716add>] xfs_file_aio_write+0x18d/0x1b0 [xfs]
[129650.722453] [<ffffffff84c41b7b>] do_sync_readv_writev+0x7b/0xd0
[129650.734768] [<ffffffff84c427cf>] compat_do_readv_writev+0xff/0x2a0
[129650.747378] [<ffffffffc0716950>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[129650.761279] [<ffffffff84c41a20>] ? do_sync_read+0xe0/0xe0
[129650.773204] [<ffffffff84b03b0f>] ? __getnstimeofday64+0x3f/0xd0
[129650.785672] [<ffffffff84c42a2e>] compat_writev+0x3e/0x80
[129650.797495] [<ffffffff84c44071>] compat_SyS_writev+0x81/0xf0
[129650.809652] [<ffffffff8517cbb0>] sysenter_dispatch+0xd/0x2b
[129650.821718] INFO: task SystemLogFlush:71425 blocked for more than 120 seconds.
[129650.835477] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129650.849754] SystemLogFlush D ffffa02e6f30d140 0 71425 71194 0x00000000
[129650.849757] Call Trace:
[129650.858514] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129650.869936] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0
[129650.882180] [<ffffffffc0710597>] ? _xfs_buf_find+0x157/0x330 [xfs]
[129650.894832] [<ffffffff8516b3a7>] __down_common+0xaa/0x104
[129650.906723] [<ffffffffc07105b0>] ? _xfs_buf_find+0x170/0x330 [xfs]
[129650.919351] [<ffffffff8516b41e>] __down+0x1d/0x1f
[129650.930493] [<ffffffff84ac8fc1>] down+0x41/0x50
[129650.941425] [<ffffffffc07103ac>] xfs_buf_lock+0x3c/0xd0 [xfs]
[129650.953571] [<ffffffffc07105b0>] _xfs_buf_find+0x170/0x330 [xfs]
[129650.966030] [<ffffffffc071079a>] xfs_buf_get_map+0x2a/0x250 [xfs]
[129650.978593] [<ffffffffc0711360>] xfs_buf_read_map+0x30/0x160 [xfs]
[129650.991620] [<ffffffffc0741db1>] xfs_trans_read_buf_map+0x211/0x400 [xfs]
[129651.005280] [<ffffffffc07010fd>] xfs_read_agi+0x9d/0x130 [xfs]
[129651.017892] [<ffffffffc07011c4>] xfs_ialloc_read_agi+0x34/0xd0 [xfs]
[129651.030754] [<ffffffffc0701bc0>] xfs_dialloc+0xf0/0x280 [xfs]
[129651.043046] [<ffffffffc0721ecd>] xfs_ialloc+0x6d/0x510 [xfs]
[129651.055058] [<ffffffffc07315c4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
[129651.068342] [<ffffffffc072447c>] xfs_dir_ialloc+0x6c/0x1f0 [xfs]
[129651.080526] [<ffffffff8516b2d2>] ? down_write+0x12/0x3d
[129651.092140] [<ffffffffc0724a8a>] xfs_create+0x48a/0x710 [xfs]
[129651.104008] [<ffffffffc0721a90>] xfs_generic_create+0xd0/0x2b0 [xfs]
[129651.116326] [<ffffffffc0721ca4>] xfs_vn_mknod+0x14/0x20 [xfs]
[129651.128033] [<ffffffffc0721cc6>] xfs_vn_mkdir+0x16/0x20 [xfs]
[129651.139648] [<ffffffff84c4f2cd>] vfs_mkdir+0xbd/0x170
[129651.150535] [<ffffffff84c5576a>] SyS_mkdirat+0xca/0x100
[129651.161509] [<ffffffff84c557b9>] SyS_mkdir+0x19/0x20
[129651.172179] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27
[129651.183774] INFO: task BgSchPool:71447 blocked for more than 120 seconds.
[129651.196206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129651.209766] BgSchPool D ffffa02e1df330c0 0 71447 71194 0x00000000
[129651.209768] Call Trace:
[129651.217892] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129651.228614] [<ffffffff8516d5ed>] rwsem_down_read_failed+0x10d/0x1a0
[129651.240766] [<ffffffffc0723360>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[129651.253765] [<ffffffff84d88778>] call_rwsem_down_read_failed+0x18/0x30
[129651.266199] [<ffffffff8516b2a0>] down_read+0x20/0x40
[129651.277057] [<ffffffffc07232ec>] xfs_ilock+0xdc/0x120 [xfs]
[129651.288547] [<ffffffffc0723360>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[129651.301425] [<ffffffffc0714de0>] xfs_dir_open+0x40/0x80 [xfs]
[129651.313035] [<ffffffff84c3fd5a>] do_dentry_open+0x1aa/0x2e0
[129651.324428] [<ffffffff84cfa152>] ? security_inode_permission+0x22/0x30
[129651.336782] [<ffffffffc0714da0>] ? xfs_file_llseek+0xa0/0xa0 [xfs]
[129651.348791] [<ffffffff84c3ff2a>] vfs_open+0x5a/0xb0
[129651.359472] [<ffffffff84c4e408>] ? may_open+0x68/0x120
[129651.370420] [<ffffffff84c50a0d>] do_last+0x1ed/0x12a0
[129651.381269] [<ffffffff84c538c7>] path_openat+0xd7/0x640
[129651.392211] [<ffffffff84b0e0d8>] ? get_futex_key+0x1c8/0x2c0
[129651.403523] [<ffffffff84c552cd>] do_filp_open+0x4d/0xb0
[129651.414265] [<ffffffff84c62a17>] ? __alloc_fd+0x47/0x170
[129651.425051] [<ffffffff84c41427>] do_sys_open+0x137/0x240
[129651.436251] [<ffffffff84c41564>] SyS_openat+0x14/0x20
[129651.446726] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27
[129651.458105] INFO: task QueryPipelineEx:71670 blocked for more than 120 seconds.
[129651.470859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[129651.484222] QueryPipelineEx D ffffa02e6fea0000 0 71670 71194 0x00000000
[129651.484224] Call Trace:
[129651.492226] [<ffffffff84d7e3c4>] ? __radix_tree_lookup+0x84/0xf0
[129651.503969] [<ffffffff8516bfb9>] schedule+0x29/0x70
[129651.514564] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0
[129651.526024] [<ffffffff84c1bff9>] ? ___slab_alloc+0x209/0x4f0
[129651.537407] [<ffffffffc071097f>] ? xfs_buf_get_map+0x20f/0x250 [xfs]
[129651.549504] [<ffffffff8516b3a7>] __down_common+0xaa/0x104
[129651.560615] [<ffffffff84d7e3c4>] ? __radix_tree_lookup+0x84/0xf0
[129651.572384] [<ffffffffc07105b0>] ? _xfs_buf_find+0x170/0x330 [xfs]
[129651.584336] [<ffffffff8516b41e>] __down+0x1d/0x1f
[129651.594803] [<ffffffff84ac8fc1>] down+0x41/0x50
[129651.605052] [<ffffffffc07103ac>] xfs_buf_lock+0x3c/0xd0 [xfs]
[129651.616565] [<ffffffffc07105b0>] _xfs_buf_find+0x170/0x330 [xfs]
[129651.628292] [<ffffffffc071079a>] xfs_buf_get_map+0x2a/0x250 [xfs]
[129651.640119] [<ffffffffc0711360>] xfs_buf_read_map+0x30/0x160 [xfs]
[129651.652030] [<ffffffffc0741db1>] xfs_trans_read_buf_map+0x211/0x400 [xfs]
[129651.664550] [<ffffffffc06d29d3>] xfs_read_agf+0x93/0x110 [xfs]
[129651.676176] [<ffffffffc06d2a9b>] xfs_alloc_read_agf+0x4b/0x1a0 [xfs]
[129651.688344] [<ffffffffc06d30ab>] xfs_alloc_fix_freelist+0x4bb/0x4e0 [xfs]
[129651.700991] [<ffffffffc0730b67>] ? kmem_alloc+0x97/0x130 [xfs]
[129651.712729] [<ffffffffc0730f07>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[129651.724872] [<ffffffff84d7e3c4>] ? __radix_tree_lookup+0x84/0xf0
[129651.736730] [<ffffffff84d7e48d>] ? radix_tree_lookup+0xd/0x10
[129651.748219] [<ffffffffc07065aa>] ? xfs_perag_get+0x2a/0xb0 [xfs]
[129651.759877] [<ffffffffc06d36af>] xfs_free_extent_fix_freelist+0x6f/0xc0 [xfs]
[129651.772719] [<ffffffffc06d3754>] xfs_free_extent+0x54/0x140 [xfs]
[129651.784491] [<ffffffffc074270a>] xfs_trans_free_extent+0x4a/0x100 [xfs]
[129651.796952] [<ffffffffc07427de>] xfs_extent_free_finish_item+0x1e/0x40 [xfs]
[129651.809659] [<ffffffffc06f1d6f>] xfs_defer_finish+0x12f/0x3a0 [xfs]
[129651.821575] [<ffffffffc07254bc>] xfs_itruncate_extents+0x11c/0x260 [xfs]
[129651.833966] [<ffffffffc07256b7>] xfs_inactive_truncate+0xb7/0x110 [xfs]
[129651.846271] [<ffffffffc0725cb8>] xfs_inactive+0x108/0x130 [xfs]
[129651.857834] [<ffffffffc072cbc5>] xfs_fs_destroy_inode+0x95/0x190 [xfs]
[129651.870032] [<ffffffff84c5fceb>] destroy_inode+0x3b/0x60
[129651.881202] [<ffffffff84c5fe25>] evict+0x115/0x180
[129651.891976] [<ffffffff84c606cc>] iput+0xfc/0x190
[129651.902279] [<ffffffff84c5477e>] do_unlinkat+0x1ae/0x2d0
[129651.913225] [<ffffffff84c55836>] SyS_unlink+0x16/0x20
[129651.924372] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27
[137921.878693] perf: interrupt took too long (7744 > 7730), lowering kernel.perf_event_max_sample_rate to 25000
[165531.276044] XFS (dm-1): Mounting V5 Filesystem
[165531.303953] XFS (dm-1): Ending clean mount
[165532.212872] XFS (dm-0): Mounting V5 Filesystem
[165532.240399] XFS (dm-0): Ending clean mount
[165533.175902] XFS (dm-0): Unmounting Filesystem
[165533.209810] XFS (dm-1): Unmounting Filesystem
[170744.629900] XFS (dm-1): Mounting V5 Filesystem
[170744.657059] XFS (dm-1): Ending clean mount
[170745.544429] XFS (dm-0): Mounting V5 Filesystem
[170745.571419] XFS (dm-0): Ending clean mount
[170746.500335] XFS (dm-0): Unmounting Filesystem
[170746.540190] XFS (dm-1): Unmounting Filesystem
@pm5
Copy link

pm5 commented Jul 26, 2023

Can I ask how this was resolved? We have a ClickHouse cluster running on XFS and we are also seeing these errors.

@filimonov
Copy link
Author

I'm not aware of reliable solutions. Try to use newer kernel or switch to ext4.
https://kb.altinity.com/altinity-kb-setup-and-maintenance/filesystems/#xfs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment