Skip to content

Instantly share code, notes, and snippets.

@smutel

smutel/front-dc-01.log

Last active May 19, 2022 15:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save smutel/790345d1ac802d679b30d119af08c7c2 to your computer and use it in GitHub Desktop.
Save smutel/790345d1ac802d679b30d119af08c7c2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
front-dc-01.log
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690380.091943] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [sshd:17129]
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.628601] rcu: INFO: rcu_sched self-detected stall on CPU
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.631659] Modules linked in: sctp drbd_transport_tcp(OE) nft_counter xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp ipt_REJECT nf_reject_ipv4 xt_multiport nft_compat nf_tables nfnetlink ext4 crc16 mbcache jbd2 fscrypto ecb crypto_simd cryptd glue_helper aes_x86_64 bochs_drm ttm drm_kms_helper drm evdev joydev sg serio_raw qemu_fw_cfg button drbd(OE) sunrpc virtio_rng rng_core ip_tables x_tables autofs4 xfs libcrc32c crc32c_generic hid_generic usbhid hid dm_mod virtio_net net_failover failover virtio_blk sr_mod cdrom uhci_hcd ehci_hcd ata_generic usbcore psmouse ata_piix libata scsi_mod usb_common virtio_pci virtio_ring i2c_piix4 virtio floppy
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.637646] rcu: 1-...!: (4 ticks this GP) idle=5da/0/0x3 softirq=77357424/77357425 fqs=0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.688239] CPU: 0 PID: 17129 Comm: sshd Tainted: G OE 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.695554] rcu: (t=6639 jiffies g=166634873 q=858)
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.703571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.709255] rcu: rcu_sched kthread starved for 6639 jiffies! g166634873 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.721518] RIP: 0010:smp_call_function_single+0xce/0xf0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.729023] rcu: RCU grace-period kthread stack dump:
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.733978] Code: 8b 4c 24 38 65 48 33 0c 25 28 00 00 00 75 34 c9 c3 48 89 d1 48 89 f2 48 89 e6 e8 6d fe ff ff 8b 54 24 18 83 e2 01 74 0b f3 90 <8b> 54 24 18 83 e2 01 75 f5 eb ca 8b 05 39 3e 4f 01 85 c0 75 88 0f
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.738731] rcu_sched I 0 10 2 0x80000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.751416] RSP: 0018:ffffaa72c09dfc00 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.758190] Call Trace:
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.764047] RAX: 0000000000000000 RBX: ffff9c7e67929de0 RCX: 0000000000000830
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.769403] __schedule+0x29f/0x840
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.774868] RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000830
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.780844] ? __switch_to_asm+0x41/0x70
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.786307] RBP: ffffaa72c09dfc50 R08: 0000000000000000 R09: 0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.790864] ? __switch_to_asm+0x35/0x70
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.796341] R10: 0000000000000002 R11: ffffaa72c0665038 R12: 0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.800797] schedule+0x28/0x80
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.806308] R13: 0000000000000200 R14: 0000000000000001 R15: ffff9c7f3bb24a38
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.810495] schedule_timeout+0x16b/0x3b0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.814312] FS: 00007f716e489e40(0000) GS:ffff9c7f3ba00000(0000) knlGS:0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.816791] ? __next_timer_interrupt+0xc0/0xc0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.820031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.822640] rcu_gp_kthread+0x40d/0x850
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.825468] CR2: 00007fff250e9ef2 CR3: 000000004ad28000 CR4: 00000000000006f0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.827907] ? call_rcu_sched+0x20/0x20
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.831082] Call Trace:
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.833536] kthread+0x112/0x130
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.835571] ? load_new_mm_cr3+0xc0/0xc0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.837773] ? kthread_bind+0x30/0x30
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.840093] ? load_new_mm_cr3+0xc0/0xc0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.842421] ret_from_fork+0x35/0x40
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.844727] ? load_new_mm_cr3+0xc0/0xc0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.847045] NMI backtrace for cpu 1
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.849286] on_each_cpu+0x28/0x60
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.851456] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G OE 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.853534] flush_tlb_kernel_range+0x48/0x90
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.857758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.859998] __purge_vmap_area_lazy+0x4d/0xc0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.867223] Call Trace:
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.869542] vm_unmap_aliases+0xe9/0x120
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.872099] <IRQ>
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.874238] change_page_attr_set_clr+0xc7/0x420
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.876547] dump_stack+0x66/0x81
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.878772] set_memory_ro+0x26/0x30
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.881440] nmi_cpu_backtrace.cold.4+0x13/0x50
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.883440] bpf_prog_select_runtime+0x28/0x110
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.886446] ? lapic_can_unplug_cpu+0x80/0x80
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.888570] bpf_prepare_filter+0x523/0x590
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.891402] nmi_trigger_cpumask_backtrace+0xf9/0x100
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.893397] bpf_prog_create_from_user+0xbb/0x110
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.896457] rcu_dump_cpu_stacks+0x9b/0xcb
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.898588] ? hardlockup_detector_perf_cleanup+0x80/0x80
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.901264] rcu_check_callbacks.cold.81+0x1db/0x335
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.903479] do_seccomp+0x25d/0x6c0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.906450] ? tick_sched_do_timer+0x60/0x60
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.908278] __x64_sys_prctl+0x4e6/0x590
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.910942] update_process_times+0x28/0x60
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.912833] do_syscall_64+0x53/0x110
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.915404] tick_sched_handle+0x22/0x60
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.917206] entry_SYSCALL_64_after_hwframe+0x44/0xa9
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.919733] tick_sched_timer+0x37/0x70
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.921923] RIP: 0033:0x7f716e83bada
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.923775] __hrtimer_run_queues+0x100/0x280
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.925515] Code: 48 8b 0d b9 03 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 9d 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 86 03 0c 00 f7 d8 64 89 01 48
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.927404] hrtimer_interrupt+0x100/0x210
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.933230] RSP: 002b:00007fff250e8888 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.935142] smp_apic_timer_interrupt+0x6a/0x140
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.938113] RAX: ffffffffffffffda RBX: 00007fff250e8890 RCX: 00007f716e83bada
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.940139] apic_timer_interrupt+0xf/0x20
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.942647] RDX: 00005624224bd040 RSI: 0000000000000002 RDI: 0000000000000016
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.944599] RIP: 0010:_raw_spin_unlock_irqrestore+0x11/0x20
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.947172] RBP: 0000562423043a80 R08: 0000000000000000 R09: 00007f716e8bbe80
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.949566] Code: d8 48 3d 90 d0 03 00 76 cc 80 4d 00 08 eb 98 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 e8 d6 04 99 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 07
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.952008] R10: 00007f716e83bada R11: 0000000000000246 R12: 00007fff250e8930
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.958254] RSP: 0018:ffff9c7f3bb03e50 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.960773] R13: 0000562423042c70 R14: 0000000000000000 R15: 0000000000000013
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.963547] RAX: 0000000000000001 RBX: ffff9c7f341c0400 RCX: 0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.969827] RDX: 000000000000e110 RSI: 0000000000000202 RDI: 0000000000000202
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.973512] RBP: ffff9c7f341c06c8 R08: 000000000000080a R09: 0000000000000001
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.976389] R10: ffff9c7f3bb03ef8 R11: 0006015669165c14 R12: ffff9c7f341c0638
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.979730] R13: 0000000000000202 R14: 0000000000000000 R15: 0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.983132] ? apic_timer_interrupt+0xa/0x20
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.985578] uhci_hub_status_data+0x63/0x220 [uhci_hcd]
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.987839] ? usb_hcd_poll_rh_status+0x1d0/0x1d0 [usbcore]
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.990311] usb_hcd_poll_rh_status+0x5c/0x1d0 [usbcore]
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.992932] ? usb_hcd_poll_rh_status+0x1d0/0x1d0 [usbcore]
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.995236] call_timer_fn+0x2b/0x130
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.997135] run_timer_softirq+0x1c7/0x3e0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690382.999561] ? kvm_clock_get_cycles+0xd/0x10
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.001821] ? ktime_get+0x3a/0xa0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.004863] __do_softirq+0xde/0x2d8
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.006968] irq_exit+0xba/0xc0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.009135] smp_apic_timer_interrupt+0x74/0x140
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.011194] apic_timer_interrupt+0xf/0x20
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.013269] </IRQ>
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.015053] RIP: 0010:native_safe_halt+0xe/0x10
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.017166] Code: ff ff 7f c3 65 48 8b 04 25 40 ac 01 00 f0 80 48 02 20 48 8b 00 a8 08 75 c4 eb 80 90 e9 07 00 00 00 0f 00 2d c6 d9 4c 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d b6 d9 4c 00 f4 c3 90 90 0f 1f 44 00
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.024230] RSP: 0018:ffffaa72c069fea8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.027415] RAX: ffffffffb753a0f0 RBX: 0000000000000001 RCX: ffffffffb7e4f310
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.030719] RDX: 00000000aec905d6 RSI: ffffffffb7e4af78 RDI: 0006015668d95314
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.033707] RBP: 0000000000000001 R08: 0006970c774490d4 R09: 0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.036620] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.039177] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.041898] ? __sched_text_end+0x7/0x7
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.044068] default_idle+0x1c/0x140
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.045916] do_idle+0x1e3/0x270
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.047917] cpu_startup_entry+0x6f/0x80
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.050228] start_secondary+0x1a4/0x200
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.052801] secondary_startup_64+0xa4/0xb0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.060538] drbd front_integ front-dc3-01: sock was shut down by peer
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.063755] drbd front_integ front-dc3-01: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.068627] drbd front_integ/0 drbd0 front-dc3-01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.073412] drbd front_integ front-dc3-01: ack_receiver terminated
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.076385] drbd front_integ front-dc3-01: Terminating ack_recv thread
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.088875] drbd front_integ front-dc3-01: Terminating sender thread
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.091690] drbd front_integ front-dc3-01: Starting sender thread (from drbd_r [651])
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.128042] drbd front_integ front-dc3-01: Connection closed
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.132010] drbd front_integ front-dc2-01: meta connection shut down by peer.
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.135012] drbd front_integ: susp-io( no -> quorum)
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.137331] drbd front_integ front-dc2-01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.141862] drbd front_integ/0 drbd0: quorum( yes -> no )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.144314] drbd front_integ/0 drbd0 front-dc2-01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.149041] drbd front_integ front-dc2-01: ack_receiver terminated
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.151850] drbd front_integ front-dc2-01: Terminating ack_recv thread
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.155370] drbd front_integ front-dc2-01: Terminating sender thread
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.158347] drbd front_integ front-dc2-01: Starting sender thread (from drbd_r [648])
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.161754] drbd front_integ/0 drbd0: helper command: /sbin/drbdadm quorum-lost
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.165307] drbd front_integ front-dc3-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.170039] drbd front_integ/0 drbd0: helper command: /sbin/drbdadm quorum-lost exit code 0
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.189299] drbd front_integ front-dc3-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.193049] drbd front_integ front-dc3-01: conn( BrokenPipe -> Unconnected )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.196025] drbd front_integ front-dc3-01: Restarting receiver thread
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.198869] drbd front_integ front-dc3-01: conn( Unconnected -> Connecting )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.201871] drbd front_integ front-dc2-01: Connection closed
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.206677] drbd front_integ front-dc2-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.211096] drbd front_integ front-dc2-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.214604] drbd front_integ front-dc2-01: conn( NetworkFailure -> Unconnected )
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.217645] drbd front_integ front-dc2-01: Restarting receiver thread
kern.log:May 16 03:32:07 front-dc1-01 kernel: [1690383.220504] drbd front_integ front-dc2-01: conn( Unconnected -> Connecting )
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.764252] drbd front_integ front-dc3-01: Handshake to peer 3 successful: Agreed network protocol version 121
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.773191] drbd front_integ front-dc3-01: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.784252] drbd front_integ front-dc3-01: Peer authenticated using 32 bytes HMAC
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.790205] drbd front_integ front-dc3-01: Starting ack_recv thread (from drbd_r [651])
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.804146] drbd front_integ: Preparing cluster-wide state change 273272025 (0->3 499/145)
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.812956] drbd front_integ front-dc2-01: Handshake to peer 1 successful: Agreed network protocol version 121
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.822131] drbd front_integ front-dc2-01: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.834900] drbd front_integ front-dc2-01: Peer authenticated using 32 bytes HMAC
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.841395] drbd front_integ front-dc2-01: Starting ack_recv thread (from drbd_r [648])
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.852167] drbd front_integ: Declined by peer front-dc3-01 (id: 3), see the kernel log there
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.858717] drbd front_integ: Aborting cluster-wide state change 273272025 (52ms) rv = -10
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.865627] drbd front_integ front-dc3-01: conn( Connecting -> Disconnecting )
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.871733] drbd front_integ front-dc3-01: ack_receiver terminated
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.877234] drbd front_integ front-dc3-01: Terminating ack_recv thread
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.883746] drbd front_integ front-dc3-01: Terminating sender thread
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.889621] drbd front_integ front-dc3-01: Starting sender thread (from drbd_r [651])
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.938001] drbd front_integ front-dc3-01: Connection closed
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.943461] drbd front_integ front-dc3-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.954120] drbd front_integ front-dc3-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.960815] drbd front_integ front-dc3-01: conn( Disconnecting -> StandAlone )
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690383.966721] drbd front_integ front-dc3-01: Terminating receiver thread
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.032118] drbd front_integ: Preparing cluster-wide state change 2357615862 (0->1 499/145)
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.060175] drbd front_integ/0 drbd0 front-dc2-01: drbd_sync_handshake:
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.066545] drbd front_integ/0 drbd0 front-dc2-01: self 914F696D337F1937:0000000000000000:32F90B65B6C364B6:10B7CCFB9050D6D0 bits:0 flags:520
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.077211] drbd front_integ/0 drbd0 front-dc2-01: peer 0005C53CD13737AF:914F696D337F1936:3F7C5C05C7099754:64E966663C58095E bits:587 flags:1120
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.087742] drbd front_integ/0 drbd0 front-dc2-01: uuid_compare()=target-use-bitmap by rule=bitmap-peer
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.096781] drbd front_integ/0 drbd0 front-dc2-01: I shall become SyncTarget, but I am primary!
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.105850] drbd front_integ: Aborting cluster-wide state change 2357615862 (72ms) rv = -27
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.114077] drbd front_integ front-dc2-01: conn( Connecting -> Disconnecting )
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.122302] drbd front_integ front-dc2-01: ack_receiver terminated
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.129624] drbd front_integ front-dc2-01: Terminating ack_recv thread
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.137161] drbd front_integ front-dc2-01: Terminating sender thread
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.142794] drbd front_integ front-dc2-01: Starting sender thread (from drbd_r [648])
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.181092] drbd front_integ front-dc2-01: Connection closed
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.187028] drbd front_integ front-dc2-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.192094] drbd front_integ front-dc2-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.195779] drbd front_integ front-dc2-01: conn( Disconnecting -> StandAlone )
kern.log:May 16 03:32:08 front-dc1-01 kernel: [1690384.198961] drbd front_integ front-dc2-01: Terminating receiver thread
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.432445] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.436972] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.439953] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.444032] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.447598] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.450975] Call Trace:
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.453582] __schedule+0x29f/0x840
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.456877] schedule+0x28/0x80
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.459610] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.462715] ? wake_up_q+0x70/0x70
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.465475] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.468414] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.471295] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.474029] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.476870] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.479432] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.482071] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.484597] worker_thread+0x30/0x390
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.486988] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.489653] kthread+0x112/0x130
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.492162] ? kthread_bind+0x30/0x30
kern.log:May 16 03:35:42 front-dc1-01 kernel: [1690597.494581] ret_from_fork+0x35/0x40
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.266826] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.270977] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.273666] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.276461] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.279276] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.281627] Call Trace:
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.283294] __schedule+0x29f/0x840
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.285697] schedule+0x28/0x80
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.287555] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.289666] ? wake_up_q+0x70/0x70
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.291408] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.293467] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.295522] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.297361] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.299328] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.301145] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.303049] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.304746] worker_thread+0x30/0x390
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.306421] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.308199] kthread+0x112/0x130
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.309781] ? kthread_bind+0x30/0x30
kern.log:May 16 03:37:43 front-dc1-01 kernel: [1690718.311477] ret_from_fork+0x35/0x40
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.105339] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.111443] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.115685] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.119801] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.123400] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.126752] Call Trace:
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.129045] __schedule+0x29f/0x840
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.133051] schedule+0x28/0x80
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.135611] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.138628] ? wake_up_q+0x70/0x70
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.141279] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.144428] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.147723] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.150614] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.153879] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.156635] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.159481] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.162269] worker_thread+0x30/0x390
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.164947] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.167874] kthread+0x112/0x130
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.170410] ? kthread_bind+0x30/0x30
kern.log:May 16 03:39:43 front-dc1-01 kernel: [1690839.172962] ret_from_fork+0x35/0x40
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.935760] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.941171] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.944973] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.948469] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.952734] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.956547] Call Trace:
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.959188] __schedule+0x29f/0x840
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.962230] schedule+0x28/0x80
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.965113] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.968532] ? wake_up_q+0x70/0x70
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.971415] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.975026] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.978403] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.981628] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.985048] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.988207] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.991348] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.994439] worker_thread+0x30/0x390
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690959.997466] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690960.000503] kthread+0x112/0x130
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690960.003242] ? kthread_bind+0x30/0x30
kern.log:May 16 03:41:44 front-dc1-01 kernel: [1690960.006232] ret_from_fork+0x35/0x40
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.770123] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.776839] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.781224] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.785698] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.789637] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.793219] Call Trace:
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.795747] __schedule+0x29f/0x840
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.798583] schedule+0x28/0x80
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.801134] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.804367] ? wake_up_q+0x70/0x70
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.807212] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.810871] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.814051] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.817095] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.820391] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.823413] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.826515] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.830061] worker_thread+0x30/0x390
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.833143] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.836337] kthread+0x112/0x130
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.839283] ? kthread_bind+0x30/0x30
kern.log:May 16 03:43:45 front-dc1-01 kernel: [1691080.842404] ret_from_fork+0x35/0x40
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.604448] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.608366] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.610835] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.613481] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.615746] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.617874] Call Trace:
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.619326] __schedule+0x29f/0x840
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.620953] schedule+0x28/0x80
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.622559] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.624519] ? wake_up_q+0x70/0x70
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.626154] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.628073] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.629973] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.631748] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.633645] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.635410] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.637299] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.638994] worker_thread+0x30/0x390
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.640616] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.642374] kthread+0x112/0x130
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.643918] ? kthread_bind+0x30/0x30
kern.log:May 16 03:45:46 front-dc1-01 kernel: [1691201.645586] ret_from_fork+0x35/0x40
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.438991] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.445946] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.450367] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.455070] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.459607] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.463364] Call Trace:
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.465923] __schedule+0x29f/0x840
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.468849] schedule+0x28/0x80
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.471666] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.475287] ? wake_up_q+0x70/0x70
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.478249] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.481925] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.485292] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.488519] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.492152] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.495263] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.498920] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.501977] worker_thread+0x30/0x390
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.505075] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.508172] kthread+0x112/0x130
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.510994] ? kthread_bind+0x30/0x30
kern.log:May 16 03:47:47 front-dc1-01 kernel: [1691322.513884] ret_from_fork+0x35/0x40
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.273266] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.277545] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.280237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.282946] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.285300] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.287504] Call Trace:
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.288981] __schedule+0x29f/0x840
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.290757] schedule+0x28/0x80
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.292343] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.294398] ? wake_up_q+0x70/0x70
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.296175] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.298260] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.300217] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.302192] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.304201] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.306081] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.307929] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.309770] worker_thread+0x30/0x390
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.311479] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.313143] kthread+0x112/0x130
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.314770] ? kthread_bind+0x30/0x30
kern.log:May 16 03:49:48 front-dc1-01 kernel: [1691443.316520] ret_from_fork+0x35/0x40
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.107689] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.111699] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.114184] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.116843] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.119225] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.121367] Call Trace:
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.122772] __schedule+0x29f/0x840
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.124474] schedule+0x28/0x80
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.125990] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.127921] ? wake_up_q+0x70/0x70
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.129580] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.131513] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.133490] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.135310] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.137253] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.138992] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.140788] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.142479] worker_thread+0x30/0x390
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.144090] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.145853] kthread+0x112/0x130
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.147339] ? kthread_bind+0x30/0x30
kern.log:May 16 03:51:48 front-dc1-01 kernel: [1691564.149015] ret_from_fork+0x35/0x40
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.942236] INFO: task kworker/0:2:15810 blocked for more than 120 seconds.
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.952460] Tainted: G OEL 4.19.0-20-amd64 #1 Debian 4.19.235-1
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.958476] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.964140] kworker/0:2 D 0 15810 2 0x80000000
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.968673] Workqueue: xfs-sync/drbd0 xfs_log_worker [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.972709] Call Trace:
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.975543] __schedule+0x29f/0x840
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.978775] schedule+0x28/0x80
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.981689] __xfs_log_force_lsn+0x14f/0x270 [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.985364] ? wake_up_q+0x70/0x70
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.988574] ? __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.992538] xfs_log_force_lsn+0x8f/0x120 [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.996127] ? xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691684.999550] __xfs_trans_commit+0x2c7/0x350 [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.003137] xfs_sync_sb+0x69/0x70 [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.006377] xfs_log_worker+0x9a/0xf0 [xfs]
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.009529] process_one_work+0x1a7/0x3a0
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.012668] worker_thread+0x30/0x390
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.015704] ? create_worker+0x1a0/0x1a0
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.018925] kthread+0x112/0x130
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.021759] ? kthread_bind+0x30/0x30
kern.log:May 16 03:53:49 front-dc1-01 kernel: [1691685.024851] ret_from_fork+0x35/0x40
Raw
front-dc2-01.log
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.751589] drbd front_integ front-dc1-01: PingAck did not arrive in time.
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.756584] drbd front_integ/0 drbd0: Would lose quorum, but using tiebreaker logic to keep
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.760130] drbd front_integ front-dc1-01: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.764097] drbd front_integ/0 drbd0: disk( UpToDate -> Consistent )
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.767533] drbd front_integ/0 drbd0 front-dc1-01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.773059] drbd front_integ front-dc1-01: ack_receiver terminated
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.776352] drbd front_integ front-dc1-01: Terminating ack_recv thread
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.780170] drbd front_integ/0 drbd0 front-dc3-01: receive_peer_dagatg(): source-set-bitmap by rule=just-created-peer
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.786053] drbd front_integ front-dc1-01: Terminating sender thread
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.789382] drbd front_integ front-dc1-01: Starting sender thread (from drbd_r [638])
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.832881] drbd front_integ front-dc1-01: Connection closed
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.836177] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.842106] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.846361] drbd front_integ front-dc1-01: conn( NetworkFailure -> Unconnected )
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.849946] drbd front_integ front-dc1-01: Restarting receiver thread
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.853150] drbd front_integ front-dc1-01: conn( Unconnected -> Connecting )
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.856615] drbd front_integ: Preparing cluster-wide state change 1289104336 (1->-1 0/0)
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.862391] drbd front_integ: State change 1289104336: primary_nodes=0, weak_nodes=0
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.865695] drbd front_integ: Committing cluster-wide state change 1289104336 (8ms)
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.868945] drbd front_integ/0 drbd0: disk( Consistent -> UpToDate )
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.872462] drbd front_integ: Preparing cluster-wide state change 2600391614 (1->-1 3/1)
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.878379] drbd front_integ: State change 2600391614: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF5
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.881848] drbd front_integ: Committing cluster-wide state change 2600391614 (8ms)
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690115.884981] drbd front_integ: role( Secondary -> Primary )
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690116.049880] drbd front_integ/0 drbd0: new current UUID: 0005C53CD13737AF weak: FFFFFFFFFFFFFFFD
kern.log:May 16 03:31:46 front-dc2-01 kernel: [1690116.055603] XFS (drbd0): Mounting V5 Filesystem
kern.log:May 16 03:31:47 front-dc2-01 kernel: [1690117.310447] XFS (drbd0): Starting recovery (logdev: internal)
kern.log:May 16 03:31:47 front-dc2-01 kernel: [1690117.331845] XFS (drbd0): Ending recovery (logdev: internal)
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.277010] drbd front_integ front-dc1-01: Handshake to peer 0 successful: Agreed network protocol version 121
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.283836] drbd front_integ front-dc1-01: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.289572] drbd front_integ front-dc3-01: Preparing remote state change 273272025
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.299428] drbd front_integ front-dc1-01: Peer authenticated using 32 bytes HMAC
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.302844] drbd front_integ front-dc1-01: Starting ack_recv thread (from drbd_r [638])
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.332426] drbd front_integ front-dc3-01: Aborting remote state change 273272025
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.503368] drbd front_integ front-dc1-01: Preparing remote state change 2357615862
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.507072] drbd front_integ: State change failed: Multiple primaries not allowed by config
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.532998] drbd front_integ/0 drbd0 front-dc1-01: drbd_sync_handshake:
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.536323] drbd front_integ/0 drbd0 front-dc1-01: self 0005C53CD13737AF:914F696D337F1936:3F7C5C05C7099754:64E966663C58095E bits:587 flags:120
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.542238] drbd front_integ/0 drbd0 front-dc1-01: peer 914F696D337F1937:0000000000000000:32F90B65B6C364B6:10B7CCFB9050D6D0 bits:0 flags:1520
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.548197] drbd front_integ/0 drbd0 front-dc1-01: uuid_compare()=source-use-bitmap by rule=bitmap-self
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.578624] drbd front_integ front-dc1-01: Aborting remote state change 2357615862
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.582227] drbd front_integ front-dc1-01: conn( Connecting -> Disconnecting )
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.585672] drbd front_integ front-dc1-01: ack_receiver terminated
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.588734] drbd front_integ front-dc1-01: Terminating ack_recv thread
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.596959] drbd front_integ front-dc1-01: Terminating sender thread
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.600002] drbd front_integ front-dc1-01: Starting sender thread (from drbd_r [638])
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.646475] drbd front_integ front-dc1-01: Connection closed
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.653235] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.664938] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.668793] drbd front_integ front-dc1-01: conn( Disconnecting -> StandAlone )
kern.log:May 16 03:32:08 front-dc2-01 kernel: [1690138.672114] drbd front_integ front-dc1-01: Terminating receiver thread
kern.log:May 16 03:32:14 front-dc2-01 kernel: [1690143.765229] XFS (drbd0): Unmounting Filesystem
kern.log:May 16 03:32:14 front-dc2-01 kernel: [1690143.850009] drbd front_integ: role( Primary -> Secondary )
Raw
front-dc3-01.log
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.912734] drbd front_integ front-dc1-01: PingAck did not arrive in time.
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.915556] drbd front_integ/0 drbd0: Would lose quorum, but using tiebreaker logic to keep
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.917257] drbd front_integ front-dc1-01: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.919126] drbd front_integ/0 drbd0 front-dc1-01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.922087] drbd front_integ front-dc1-01: ack_receiver terminated
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.923721] drbd front_integ front-dc1-01: Terminating ack_recv thread
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.925689] drbd front_integ front-dc1-01: Terminating sender thread
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.927297] drbd front_integ front-dc1-01: Starting sender thread (from drbd_r [525])
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.967658] drbd front_integ front-dc1-01: Connection closed
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.969201] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.972467] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.974301] drbd front_integ front-dc1-01: conn( NetworkFailure -> Unconnected )
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.975964] drbd front_integ front-dc1-01: Restarting receiver thread
kern.log:May 16 03:31:45 front-dc3-01 kernel: [1689806.977484] drbd front_integ front-dc1-01: conn( Unconnected -> Connecting )
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.517696] drbd front_integ/0 drbd0 front-dc2-01: pdsk( UpToDate -> Consistent )
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.594151] drbd front_integ front-dc2-01: Reconciliation resync because 'front-dc1-01' disappeared. (o=-991161)
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.597542] drbd front_integ front-dc2-01: Preparing remote state change 1289104336
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.609470] drbd front_integ front-dc2-01: Committing remote state change 1289104336 (primary_nodes=0)
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.611307] drbd front_integ/0 drbd0 front-dc2-01: pdsk( Consistent -> UpToDate )
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.613546] drbd front_integ front-dc2-01: Preparing remote state change 2600391614
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.626142] drbd front_integ front-dc2-01: Committing remote state change 2600391614 (primary_nodes=2)
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.628063] drbd front_integ front-dc2-01: peer( Secondary -> Primary )
kern.log:May 16 03:31:46 front-dc3-01 kernel: [1689807.791018] drbd front_integ/0 drbd0: receiver updated UUIDs to exposed data uuid: 0005C53CD13737AF
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689829.964458] drbd front_integ front-dc1-01: Handshake to peer 0 successful: Agreed network protocol version 121
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689829.968412] drbd front_integ front-dc1-01: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689829.983728] drbd front_integ front-dc1-01: Peer authenticated using 32 bytes HMAC
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689829.985686] drbd front_integ front-dc1-01: Starting ack_recv thread (from drbd_r [525])
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.010935] drbd front_integ front-dc1-01: Preparing remote state change 273272025
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.065861] drbd front_integ front-dc1-01: Aborting remote state change 273272025
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.067783] drbd front_integ front-dc1-01: conn( Connecting -> Disconnecting )
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.069698] drbd front_integ front-dc1-01: ack_receiver terminated
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.071340] drbd front_integ front-dc1-01: Terminating ack_recv thread
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.073353] drbd front_integ front-dc1-01: Terminating sender thread
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.075004] drbd front_integ front-dc1-01: Starting sender thread (from drbd_r [525])
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.107275] drbd front_integ front-dc1-01: Connection closed
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.108979] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.112057] drbd front_integ front-dc1-01: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.113860] drbd front_integ front-dc1-01: conn( Disconnecting -> StandAlone )
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.115585] drbd front_integ front-dc1-01: Terminating receiver thread
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.248741] drbd front_integ front-dc2-01: Preparing remote state change 2357615862
kern.log:May 16 03:32:08 front-dc3-01 kernel: [1689830.317920] drbd front_integ front-dc2-01: Aborting remote state change 2357615862
kern.log:May 16 03:32:14 front-dc3-01 kernel: [1689835.592353] drbd front_integ front-dc2-01: peer( Primary -> Secondary )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment