Created
September 16, 2021 15:33
-
-
Save filimonov/85b894268f978c2ccc18ea69bae5adbd to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[51140.383890] perf: interrupt took too long (2519 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 | |
[59014.200296] perf: interrupt took too long (3150 > 3148), lowering kernel.perf_event_max_sample_rate to 63000 | |
[59445.940213] perf: interrupt took too long (3939 > 3937), lowering kernel.perf_event_max_sample_rate to 50000 | |
[79341.838396] XFS (dm-1): Mounting V5 Filesystem | |
[79341.862314] XFS (dm-1): Ending clean mount | |
[79342.489941] XFS (dm-0): Mounting V5 Filesystem | |
[79342.510930] XFS (dm-0): Ending clean mount | |
[79343.419111] XFS (dm-0): Unmounting Filesystem | |
[79343.451992] XFS (dm-1): Unmounting Filesystem | |
[85991.320353] XFS (dm-1): Mounting V5 Filesystem | |
[85991.342594] XFS (dm-1): Ending clean mount | |
[85991.982472] XFS (dm-0): Mounting V5 Filesystem | |
[85992.004335] XFS (dm-0): Ending clean mount | |
[85992.907717] XFS (dm-0): Unmounting Filesystem | |
[85992.935703] XFS (dm-1): Unmounting Filesystem | |
[102694.188000] perf: interrupt took too long (4941 > 4923), lowering kernel.perf_event_max_sample_rate to 40000 | |
[126340.883673] CPU57: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883675] CPU49: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883676] CPU51: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883677] CPU15: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883678] CPU43: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883679] CPU7: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883680] CPU61: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883682] CPU37: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883683] CPU19: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883684] CPU55: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883685] CPU1: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883686] CPU63: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883687] CPU27: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883688] CPU21: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883689] CPU69: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883690] CPU33: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883691] CPU23: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883692] CPU59: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883693] CPU13: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883695] CPU41: Core temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883695] CPU5: Core temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883697] CPU53: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883697] CPU17: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883699] CPU67: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883700] CPU31: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883701] CPU39: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883702] CPU3: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883703] CPU71: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883704] CPU35: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883706] CPU29: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883706] CPU65: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883708] CPU45: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883708] CPU9: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883709] CPU25: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883710] CPU47: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883711] CPU11: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883712] CPU41: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.883712] CPU5: Package temperature above threshold, cpu clock throttled (total events = 1) | |
[126340.885707] CPU43: Package temperature/speed normal | |
[126340.885709] CPU47: Package temperature/speed normal | |
[126340.885709] CPU13: Package temperature/speed normal | |
[126340.885710] CPU11: Package temperature/speed normal | |
[126340.885711] CPU53: Package temperature/speed normal | |
[126340.885712] CPU71: Package temperature/speed normal | |
[126340.885712] CPU15: Package temperature/speed normal | |
[126340.885713] CPU21: Package temperature/speed normal | |
[126340.885714] CPU59: Package temperature/speed normal | |
[126340.885714] CPU23: Package temperature/speed normal | |
[126340.885715] CPU69: Package temperature/speed normal | |
[126340.885715] CPU33: Package temperature/speed normal | |
[126340.885716] CPU41: Core temperature/speed normal | |
[126340.885717] CPU5: Core temperature/speed normal | |
[126340.885717] CPU27: Package temperature/speed normal | |
[126340.885718] CPU63: Package temperature/speed normal | |
[126340.885719] CPU55: Package temperature/speed normal | |
[126340.885719] CPU31: Package temperature/speed normal | |
[126340.885720] CPU67: Package temperature/speed normal | |
[126340.885720] CPU65: Package temperature/speed normal | |
[126340.885721] CPU29: Package temperature/speed normal | |
[126340.885722] CPU49: Package temperature/speed normal | |
[126340.885722] CPU51: Package temperature/speed normal | |
[126340.885723] CPU3: Package temperature/speed normal | |
[126340.885724] CPU9: Package temperature/speed normal | |
[126340.885725] CPU35: Package temperature/speed normal | |
[126340.885725] CPU17: Package temperature/speed normal | |
[126340.885726] CPU19: Package temperature/speed normal | |
[126340.885727] CPU45: Package temperature/speed normal | |
[126340.885727] CPU25: Package temperature/speed normal | |
[126340.885728] CPU7: Package temperature/speed normal | |
[126340.885728] CPU37: Package temperature/speed normal | |
[126340.885729] CPU1: Package temperature/speed normal | |
[126340.885730] CPU39: Package temperature/speed normal | |
[126340.885730] CPU61: Package temperature/speed normal | |
[126340.885731] CPU41: Package temperature/speed normal | |
[126340.885731] CPU5: Package temperature/speed normal | |
[126341.271473] CPU57: Package temperature/speed normal | |
[127490.659951] perf: interrupt took too long (6184 > 6176), lowering kernel.perf_event_max_sample_rate to 32000 | |
[129648.987293] INFO: task systemd:1 blocked for more than 120 seconds. | |
[129648.998027] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129649.010457] systemd D ffff9fa1adb78000 0 1 0 0x00000000 | |
[129649.010466] Call Trace: | |
[129649.017564] [<ffffffff84d50ea5>] ? blk_mq_start_request+0x75/0x160 | |
[129649.028625] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129649.038449] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0 | |
[129649.049177] [<ffffffff8516c36d>] wait_for_completion+0xfd/0x140 | |
[129649.060141] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20 | |
[129649.070502] [<ffffffff84abb93d>] flush_work+0xfd/0x190 | |
[129649.080321] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90 | |
[129649.090989] [<ffffffffc07367ea>] xlog_cil_force_lsn+0x8a/0x210 [xfs] | |
[129649.102243] [<ffffffffc0734b90>] _xfs_log_force_lsn+0x80/0x340 [xfs] | |
[129649.113546] [<ffffffff84d48a64>] ? blk_finish_plug+0x14/0x40 | |
[129649.124260] [<ffffffffc07229bc>] ? __xfs_iunpin_wait+0x9c/0x150 [xfs] | |
[129649.135702] [<ffffffffc0734e84>] xfs_log_force_lsn+0x34/0x70 [xfs] | |
[129649.146794] [<ffffffffc0725cf9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] | |
[129649.157933] [<ffffffffc07229bc>] __xfs_iunpin_wait+0x9c/0x150 [xfs] | |
[129649.169195] [<ffffffff84ac4230>] ? wake_bit_function+0x40/0x40 | |
[129649.180125] [<ffffffffc0725cf9>] xfs_iunpin_wait+0x19/0x20 [xfs] | |
[129649.191145] [<ffffffffc0719543>] xfs_reclaim_inode+0x143/0x360 [xfs] | |
[129649.202410] [<ffffffffc07199c7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] | |
[129649.214108] [<ffffffffc071aba3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] | |
[129649.225690] [<ffffffffc072b1f5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] | |
[129649.237822] [<ffffffff84c451d6>] prune_super+0xf6/0x190 | |
[129649.248285] [<ffffffff84bcb375>] shrink_slab+0x175/0x340 | |
[129649.258919] [<ffffffff84c385d1>] ? vmpressure+0x21/0x90 | |
[129649.269536] [<ffffffff84bce632>] do_try_to_free_pages+0x3c2/0x4e0 | |
[129649.281135] [<ffffffff84bce84c>] try_to_free_pages+0xfc/0x180 | |
[129649.292273] [<ffffffff8516158c>] __alloc_pages_slowpath+0x457/0x724 | |
[129649.303886] [<ffffffff84bc2744>] __alloc_pages_nodemask+0x404/0x420 | |
[129649.315359] [<ffffffff84a960fd>] copy_process+0x1dd/0x1a40 | |
[129649.326132] [<ffffffff84a97b11>] do_fork+0x91/0x320 | |
[129649.336368] [<ffffffff84cfa62c>] ? security_file_fcntl+0x1c/0x20 | |
[129649.347857] [<ffffffff84c5610f>] ? SyS_fcntl+0xef/0x6f0 | |
[129649.358651] [<ffffffff84a97e26>] SyS_clone+0x16/0x20 | |
[129649.369056] [<ffffffff851791b4>] stub_clone+0x44/0x70 | |
[129649.379445] [<ffffffff85178ddb>] ? system_call_fastpath+0x22/0x27 | |
[129649.391010] INFO: task kswapd0:390 blocked for more than 120 seconds. | |
[129649.402940] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129649.416485] kswapd0 D ffffa02e7e0bc100 0 390 2 0x00000000 | |
[129649.416488] Call Trace: | |
[129649.424700] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129649.435408] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0 | |
[129649.446883] [<ffffffff8516c36d>] wait_for_completion+0xfd/0x140 | |
[129649.458629] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20 | |
[129649.469992] [<ffffffff84abb93d>] flush_work+0xfd/0x190 | |
[129649.481094] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90 | |
[129649.492805] [<ffffffffc07367ea>] xlog_cil_force_lsn+0x8a/0x210 [xfs] | |
[129649.504940] [<ffffffffc0734b90>] _xfs_log_force_lsn+0x80/0x340 [xfs] | |
[129649.516844] [<ffffffff84d48a64>] ? blk_finish_plug+0x14/0x40 | |
[129649.527987] [<ffffffffc07229bc>] ? __xfs_iunpin_wait+0x9c/0x150 [xfs] | |
[129649.539830] [<ffffffffc0734e84>] xfs_log_force_lsn+0x34/0x70 [xfs] | |
[129649.551384] [<ffffffffc0725cf9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] | |
[129649.562905] [<ffffffffc07229bc>] __xfs_iunpin_wait+0x9c/0x150 [xfs] | |
[129649.574477] [<ffffffff84ac4230>] ? wake_bit_function+0x40/0x40 | |
[129649.585656] [<ffffffffc0725cf9>] xfs_iunpin_wait+0x19/0x20 [xfs] | |
[129649.597041] [<ffffffffc0719543>] xfs_reclaim_inode+0x143/0x360 [xfs] | |
[129649.608792] [<ffffffffc07199c7>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] | |
[129649.620884] [<ffffffffc071aba3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] | |
[129649.632751] [<ffffffffc072b1f5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] | |
[129649.645115] [<ffffffff84c451d6>] prune_super+0xf6/0x190 | |
[129649.655698] [<ffffffff84bcb375>] shrink_slab+0x175/0x340 | |
[129649.666333] [<ffffffff84c38637>] ? vmpressure+0x87/0x90 | |
[129649.676836] [<ffffffff84bcf171>] balance_pgdat+0x4b1/0x5e0 | |
[129649.687620] [<ffffffff84bcf413>] kswapd+0x173/0x440 | |
[129649.697813] [<ffffffff84ac4170>] ? wake_up_atomic_t+0x30/0x30 | |
[129649.708864] [<ffffffff84bcf2a0>] ? balance_pgdat+0x5e0/0x5e0 | |
[129649.719810] [<ffffffff84ac30a1>] kthread+0xd1/0xe0 | |
[129649.729852] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40 | |
[129649.741153] [<ffffffff85178c1d>] ret_from_fork_nospec_begin+0x7/0x21 | |
[129649.752870] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40 | |
[129649.764249] INFO: task xfsaild/dm-6:40642 blocked for more than 120 seconds. | |
[129649.776615] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129649.789896] xfsaild/dm-6 D ffffa0be79534100 0 40642 2 0x00000000 | |
[129649.789899] Call Trace: | |
[129649.798045] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90 | |
[129649.809522] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129649.820026] [<ffffffffc0736942>] xlog_cil_force_lsn+0x1e2/0x210 [xfs] | |
[129649.832230] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20 | |
[129649.843449] [<ffffffffc0734765>] _xfs_log_force+0x85/0x2c0 [xfs] | |
[129649.855161] [<ffffffff84aaad50>] ? __internal_add_timer+0x130/0x130 | |
[129649.867195] [<ffffffffc0740aac>] ? xfsaild+0x16c/0x6f0 [xfs] | |
[129649.878681] [<ffffffffc07349cc>] xfs_log_force+0x2c/0x70 [xfs] | |
[129649.890392] [<ffffffffc0740940>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] | |
[129649.903541] [<ffffffffc0740aac>] xfsaild+0x16c/0x6f0 [xfs] | |
[129649.914886] [<ffffffffc0740940>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] | |
[129649.927945] [<ffffffff84ac30a1>] kthread+0xd1/0xe0 | |
[129649.938584] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40 | |
[129649.950506] [<ffffffff85178c1d>] ret_from_fork_nospec_begin+0x7/0x21 | |
[129649.962586] [<ffffffff84ac2fd0>] ? insert_kthread_work+0x40/0x40 | |
[129649.974330] INFO: task systemd-logind:40802 blocked for more than 120 seconds. | |
[129649.987306] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129650.001000] systemd-logind D ffffa02e68b68000 0 40802 1 0x00000000 | |
[129650.001002] Call Trace: | |
[129650.009297] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129650.020182] [<ffffffff8516d8a5>] rwsem_down_write_failed+0x225/0x3a0 | |
[129650.032595] [<ffffffff84d77fc2>] ? get_from_free_list+0x42/0x50 | |
[129650.044561] [<ffffffff84d887a7>] call_rwsem_down_write_failed+0x17/0x30 | |
[129650.057279] [<ffffffff84d79260>] ? ida_get_new_above+0x230/0x2a0 | |
[129650.069372] [<ffffffff8516b2ed>] down_write+0x2d/0x3d | |
[129650.080479] [<ffffffff84bca831>] register_shrinker+0x21/0x50 | |
[129650.092218] [<ffffffff84c46429>] sget_userns+0x489/0x4c0 | |
[129650.103616] [<ffffffff84c45380>] ? get_anon_bdev+0x110/0x110 | |
[129650.115387] [<ffffffff84c464dd>] sget+0x7d/0xb0 | |
[129650.125974] [<ffffffff84c45380>] ? get_anon_bdev+0x110/0x110 | |
[129650.137700] [<ffffffff84bd1330>] ? shmem_rename+0x20/0x20 | |
[129650.149133] [<ffffffff84c46540>] mount_nodev+0x30/0xb0 | |
[129650.160310] [<ffffffff84bd0fb8>] shmem_mount+0x18/0x20 | |
[129650.171477] [<ffffffff84c470de>] mount_fs+0x3e/0x1b0 | |
[129650.182440] [<ffffffff84c64d07>] vfs_kern_mount+0x67/0x110 | |
[129650.193886] [<ffffffff84c6732f>] do_mount+0x1ef/0xce0 | |
[129650.204895] [<ffffffff84c67065>] ? copy_mount_options+0xc5/0x170 | |
[129650.216916] [<ffffffff84c68163>] SyS_mount+0x83/0xd0 | |
[129650.227871] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27 | |
[129650.239862] INFO: task console:60319 blocked for more than 120 seconds. | |
[129650.252502] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129650.266498] console D ffffa0be2fa3b0c0 0 60319 60311 0x20020000 | |
[129650.266501] Call Trace: | |
[129650.275091] [<ffffffff84ab8730>] ? move_linked_works+0x90/0x90 | |
[129650.287279] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129650.298476] [<ffffffffc0736942>] xlog_cil_force_lsn+0x1e2/0x210 [xfs] | |
[129650.311263] [<ffffffff84ad7e60>] ? wake_up_state+0x20/0x20 | |
[129650.323048] [<ffffffffc0734b90>] _xfs_log_force_lsn+0x80/0x340 [xfs] | |
[129650.335626] [<ffffffff84bb6f6f>] ? filemap_fdatawait_range+0x1f/0x30 | |
[129650.348135] [<ffffffff8516b292>] ? down_read+0x12/0x40 | |
[129650.359385] [<ffffffffc0714f5d>] xfs_file_fsync+0xfd/0x1c0 [xfs] | |
[129650.371519] [<ffffffff84c77207>] do_fsync+0x67/0xb0 | |
[129650.383675] [<ffffffff84c774f0>] SyS_fsync+0x10/0x20 | |
[129650.394706] [<ffffffff8517cbb0>] sysenter_dispatch+0xd/0x2b | |
[129650.406469] INFO: task console:62065 blocked for more than 120 seconds. | |
[129650.419099] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129650.433357] console D ffffa02e62751040 0 62065 60311 0x20020000 | |
[129650.433359] Call Trace: | |
[129650.441948] [<ffffffff8516ced9>] schedule_preempt_disabled+0x29/0x70 | |
[129650.454598] [<ffffffff8516ae57>] __mutex_lock_slowpath+0xc7/0x1d0 | |
[129650.466960] [<ffffffff8516a23f>] mutex_lock+0x1f/0x2f | |
[129650.478215] [<ffffffff84c44057>] compat_SyS_writev+0x67/0xf0 | |
[129650.490048] [<ffffffff8517cbb0>] sysenter_dispatch+0xd/0x2b | |
[129650.501787] INFO: task console:62078 blocked for more than 120 seconds. | |
[129650.514554] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129650.528628] console D ffffa02e62752080 0 62078 60311 0x20020000 | |
[129650.528630] Call Trace: | |
[129650.537212] [<ffffffff84c1da62>] ? kmem_cache_alloc+0x1c2/0x1f0 | |
[129650.549406] [<ffffffffc0730f07>] ? kmem_zone_alloc+0x97/0x130 [xfs] | |
[129650.561841] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129650.572865] [<ffffffff8516d8a5>] rwsem_down_write_failed+0x225/0x3a0 | |
[129650.585424] [<ffffffffc07315c4>] ? xlog_grant_head_check+0x54/0x100 [xfs] | |
[129650.598427] [<ffffffffc072057d>] ? xfs_vn_update_time+0xcd/0x150 [xfs] | |
[129650.611177] [<ffffffff84d887a7>] call_rwsem_down_write_failed+0x17/0x30 | |
[129650.624015] [<ffffffff8516b2ed>] down_write+0x2d/0x3d | |
[129650.635283] [<ffffffffc07232d4>] xfs_ilock+0xc4/0x120 [xfs] | |
[129650.647066] [<ffffffffc072057d>] xfs_vn_update_time+0xcd/0x150 [xfs] | |
[129650.659646] [<ffffffff84c5f4e8>] update_time+0x28/0xd0 | |
[129650.671022] [<ffffffff84c5f630>] file_update_time+0xa0/0xf0 | |
[129650.682826] [<ffffffffc071612d>] xfs_file_aio_write_checks+0x18d/0x200 [xfs] | |
[129650.696194] [<ffffffffc071675a>] xfs_file_buffered_aio_write+0xca/0x2c0 [xfs] | |
[129650.709678] [<ffffffffc0716add>] xfs_file_aio_write+0x18d/0x1b0 [xfs] | |
[129650.722453] [<ffffffff84c41b7b>] do_sync_readv_writev+0x7b/0xd0 | |
[129650.734768] [<ffffffff84c427cf>] compat_do_readv_writev+0xff/0x2a0 | |
[129650.747378] [<ffffffffc0716950>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs] | |
[129650.761279] [<ffffffff84c41a20>] ? do_sync_read+0xe0/0xe0 | |
[129650.773204] [<ffffffff84b03b0f>] ? __getnstimeofday64+0x3f/0xd0 | |
[129650.785672] [<ffffffff84c42a2e>] compat_writev+0x3e/0x80 | |
[129650.797495] [<ffffffff84c44071>] compat_SyS_writev+0x81/0xf0 | |
[129650.809652] [<ffffffff8517cbb0>] sysenter_dispatch+0xd/0x2b | |
[129650.821718] INFO: task SystemLogFlush:71425 blocked for more than 120 seconds. | |
[129650.835477] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129650.849754] SystemLogFlush D ffffa02e6f30d140 0 71425 71194 0x00000000 | |
[129650.849757] Call Trace: | |
[129650.858514] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129650.869936] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0 | |
[129650.882180] [<ffffffffc0710597>] ? _xfs_buf_find+0x157/0x330 [xfs] | |
[129650.894832] [<ffffffff8516b3a7>] __down_common+0xaa/0x104 | |
[129650.906723] [<ffffffffc07105b0>] ? _xfs_buf_find+0x170/0x330 [xfs] | |
[129650.919351] [<ffffffff8516b41e>] __down+0x1d/0x1f | |
[129650.930493] [<ffffffff84ac8fc1>] down+0x41/0x50 | |
[129650.941425] [<ffffffffc07103ac>] xfs_buf_lock+0x3c/0xd0 [xfs] | |
[129650.953571] [<ffffffffc07105b0>] _xfs_buf_find+0x170/0x330 [xfs] | |
[129650.966030] [<ffffffffc071079a>] xfs_buf_get_map+0x2a/0x250 [xfs] | |
[129650.978593] [<ffffffffc0711360>] xfs_buf_read_map+0x30/0x160 [xfs] | |
[129650.991620] [<ffffffffc0741db1>] xfs_trans_read_buf_map+0x211/0x400 [xfs] | |
[129651.005280] [<ffffffffc07010fd>] xfs_read_agi+0x9d/0x130 [xfs] | |
[129651.017892] [<ffffffffc07011c4>] xfs_ialloc_read_agi+0x34/0xd0 [xfs] | |
[129651.030754] [<ffffffffc0701bc0>] xfs_dialloc+0xf0/0x280 [xfs] | |
[129651.043046] [<ffffffffc0721ecd>] xfs_ialloc+0x6d/0x510 [xfs] | |
[129651.055058] [<ffffffffc07315c4>] ? xlog_grant_head_check+0x54/0x100 [xfs] | |
[129651.068342] [<ffffffffc072447c>] xfs_dir_ialloc+0x6c/0x1f0 [xfs] | |
[129651.080526] [<ffffffff8516b2d2>] ? down_write+0x12/0x3d | |
[129651.092140] [<ffffffffc0724a8a>] xfs_create+0x48a/0x710 [xfs] | |
[129651.104008] [<ffffffffc0721a90>] xfs_generic_create+0xd0/0x2b0 [xfs] | |
[129651.116326] [<ffffffffc0721ca4>] xfs_vn_mknod+0x14/0x20 [xfs] | |
[129651.128033] [<ffffffffc0721cc6>] xfs_vn_mkdir+0x16/0x20 [xfs] | |
[129651.139648] [<ffffffff84c4f2cd>] vfs_mkdir+0xbd/0x170 | |
[129651.150535] [<ffffffff84c5576a>] SyS_mkdirat+0xca/0x100 | |
[129651.161509] [<ffffffff84c557b9>] SyS_mkdir+0x19/0x20 | |
[129651.172179] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27 | |
[129651.183774] INFO: task BgSchPool:71447 blocked for more than 120 seconds. | |
[129651.196206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129651.209766] BgSchPool D ffffa02e1df330c0 0 71447 71194 0x00000000 | |
[129651.209768] Call Trace: | |
[129651.217892] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129651.228614] [<ffffffff8516d5ed>] rwsem_down_read_failed+0x10d/0x1a0 | |
[129651.240766] [<ffffffffc0723360>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs] | |
[129651.253765] [<ffffffff84d88778>] call_rwsem_down_read_failed+0x18/0x30 | |
[129651.266199] [<ffffffff8516b2a0>] down_read+0x20/0x40 | |
[129651.277057] [<ffffffffc07232ec>] xfs_ilock+0xdc/0x120 [xfs] | |
[129651.288547] [<ffffffffc0723360>] xfs_ilock_data_map_shared+0x30/0x40 [xfs] | |
[129651.301425] [<ffffffffc0714de0>] xfs_dir_open+0x40/0x80 [xfs] | |
[129651.313035] [<ffffffff84c3fd5a>] do_dentry_open+0x1aa/0x2e0 | |
[129651.324428] [<ffffffff84cfa152>] ? security_inode_permission+0x22/0x30 | |
[129651.336782] [<ffffffffc0714da0>] ? xfs_file_llseek+0xa0/0xa0 [xfs] | |
[129651.348791] [<ffffffff84c3ff2a>] vfs_open+0x5a/0xb0 | |
[129651.359472] [<ffffffff84c4e408>] ? may_open+0x68/0x120 | |
[129651.370420] [<ffffffff84c50a0d>] do_last+0x1ed/0x12a0 | |
[129651.381269] [<ffffffff84c538c7>] path_openat+0xd7/0x640 | |
[129651.392211] [<ffffffff84b0e0d8>] ? get_futex_key+0x1c8/0x2c0 | |
[129651.403523] [<ffffffff84c552cd>] do_filp_open+0x4d/0xb0 | |
[129651.414265] [<ffffffff84c62a17>] ? __alloc_fd+0x47/0x170 | |
[129651.425051] [<ffffffff84c41427>] do_sys_open+0x137/0x240 | |
[129651.436251] [<ffffffff84c41564>] SyS_openat+0x14/0x20 | |
[129651.446726] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27 | |
[129651.458105] INFO: task QueryPipelineEx:71670 blocked for more than 120 seconds. | |
[129651.470859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. | |
[129651.484222] QueryPipelineEx D ffffa02e6fea0000 0 71670 71194 0x00000000 | |
[129651.484224] Call Trace: | |
[129651.492226] [<ffffffff84d7e3c4>] ? __radix_tree_lookup+0x84/0xf0 | |
[129651.503969] [<ffffffff8516bfb9>] schedule+0x29/0x70 | |
[129651.514564] [<ffffffff85169ac1>] schedule_timeout+0x221/0x2d0 | |
[129651.526024] [<ffffffff84c1bff9>] ? ___slab_alloc+0x209/0x4f0 | |
[129651.537407] [<ffffffffc071097f>] ? xfs_buf_get_map+0x20f/0x250 [xfs] | |
[129651.549504] [<ffffffff8516b3a7>] __down_common+0xaa/0x104 | |
[129651.560615] [<ffffffff84d7e3c4>] ? __radix_tree_lookup+0x84/0xf0 | |
[129651.572384] [<ffffffffc07105b0>] ? _xfs_buf_find+0x170/0x330 [xfs] | |
[129651.584336] [<ffffffff8516b41e>] __down+0x1d/0x1f | |
[129651.594803] [<ffffffff84ac8fc1>] down+0x41/0x50 | |
[129651.605052] [<ffffffffc07103ac>] xfs_buf_lock+0x3c/0xd0 [xfs] | |
[129651.616565] [<ffffffffc07105b0>] _xfs_buf_find+0x170/0x330 [xfs] | |
[129651.628292] [<ffffffffc071079a>] xfs_buf_get_map+0x2a/0x250 [xfs] | |
[129651.640119] [<ffffffffc0711360>] xfs_buf_read_map+0x30/0x160 [xfs] | |
[129651.652030] [<ffffffffc0741db1>] xfs_trans_read_buf_map+0x211/0x400 [xfs] | |
[129651.664550] [<ffffffffc06d29d3>] xfs_read_agf+0x93/0x110 [xfs] | |
[129651.676176] [<ffffffffc06d2a9b>] xfs_alloc_read_agf+0x4b/0x1a0 [xfs] | |
[129651.688344] [<ffffffffc06d30ab>] xfs_alloc_fix_freelist+0x4bb/0x4e0 [xfs] | |
[129651.700991] [<ffffffffc0730b67>] ? kmem_alloc+0x97/0x130 [xfs] | |
[129651.712729] [<ffffffffc0730f07>] ? kmem_zone_alloc+0x97/0x130 [xfs] | |
[129651.724872] [<ffffffff84d7e3c4>] ? __radix_tree_lookup+0x84/0xf0 | |
[129651.736730] [<ffffffff84d7e48d>] ? radix_tree_lookup+0xd/0x10 | |
[129651.748219] [<ffffffffc07065aa>] ? xfs_perag_get+0x2a/0xb0 [xfs] | |
[129651.759877] [<ffffffffc06d36af>] xfs_free_extent_fix_freelist+0x6f/0xc0 [xfs] | |
[129651.772719] [<ffffffffc06d3754>] xfs_free_extent+0x54/0x140 [xfs] | |
[129651.784491] [<ffffffffc074270a>] xfs_trans_free_extent+0x4a/0x100 [xfs] | |
[129651.796952] [<ffffffffc07427de>] xfs_extent_free_finish_item+0x1e/0x40 [xfs] | |
[129651.809659] [<ffffffffc06f1d6f>] xfs_defer_finish+0x12f/0x3a0 [xfs] | |
[129651.821575] [<ffffffffc07254bc>] xfs_itruncate_extents+0x11c/0x260 [xfs] | |
[129651.833966] [<ffffffffc07256b7>] xfs_inactive_truncate+0xb7/0x110 [xfs] | |
[129651.846271] [<ffffffffc0725cb8>] xfs_inactive+0x108/0x130 [xfs] | |
[129651.857834] [<ffffffffc072cbc5>] xfs_fs_destroy_inode+0x95/0x190 [xfs] | |
[129651.870032] [<ffffffff84c5fceb>] destroy_inode+0x3b/0x60 | |
[129651.881202] [<ffffffff84c5fe25>] evict+0x115/0x180 | |
[129651.891976] [<ffffffff84c606cc>] iput+0xfc/0x190 | |
[129651.902279] [<ffffffff84c5477e>] do_unlinkat+0x1ae/0x2d0 | |
[129651.913225] [<ffffffff84c55836>] SyS_unlink+0x16/0x20 | |
[129651.924372] [<ffffffff85178ddb>] system_call_fastpath+0x22/0x27 | |
[137921.878693] perf: interrupt took too long (7744 > 7730), lowering kernel.perf_event_max_sample_rate to 25000 | |
[165531.276044] XFS (dm-1): Mounting V5 Filesystem | |
[165531.303953] XFS (dm-1): Ending clean mount | |
[165532.212872] XFS (dm-0): Mounting V5 Filesystem | |
[165532.240399] XFS (dm-0): Ending clean mount | |
[165533.175902] XFS (dm-0): Unmounting Filesystem | |
[165533.209810] XFS (dm-1): Unmounting Filesystem | |
[170744.629900] XFS (dm-1): Mounting V5 Filesystem | |
[170744.657059] XFS (dm-1): Ending clean mount | |
[170745.544429] XFS (dm-0): Mounting V5 Filesystem | |
[170745.571419] XFS (dm-0): Ending clean mount | |
[170746.500335] XFS (dm-0): Unmounting Filesystem | |
[170746.540190] XFS (dm-1): Unmounting Filesystem |
I'm not aware of reliable solutions. Try to use newer kernel or switch to ext4.
https://kb.altinity.com/altinity-kb-setup-and-maintenance/filesystems/#xfs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Can I ask how this was resolved? We have a ClickHouse cluster running on XFS and we are also seeing these errors.