Skip to content

Instantly share code, notes, and snippets.

@sodar

sodar/rr-with-dpdk.log

Last active August 14, 2020 09:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sodar/15c75e2796b2e2d9a36f74a689467bd4 to your computer and use it in GitHub Desktop.
Save sodar/15c75e2796b2e2d9a36f74a689467bd4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
rr-with-dpdk.log
[FATAL /rr/rr/src/ReplaySession.cc:1267:flush_syscallbuf()]
(task 13694 (rec:13688) at time 3023)
-> Assertion `t->stop_sig() == SIGSEGV && r.ip() == t->vm()->do_breakpoint_fault_addr()' failed to hold. Replay got unexpected signal (or none) 11
Tail of trace dump:
{
real_time:1465.816645 global_time:3003, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:13688, ticks:7084892
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x2 rsi:0x7ffcf7b76a30 rdi:0xffffff9c rbp:0x2 rsp:0x681ffde0 r8:0x0 r9:0x7ffcf7b75787 r10:0x0 r11:0x246 r12:0x400630 r13:0x7ffcf7b78f90 r14:0x7ffcf7b76a30 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.818586 global_time:3004, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:13688, ticks:7084892
rax:0x11 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x2 rsi:0x7ffcf7b76a30 rdi:0xffffff9c rbp:0x2 rsp:0x681ffde0 r8:0x0 r9:0x7ffcf7b75787 r10:0x0 r11:0x246 r12:0x400630 r13:0x7ffcf7b78f90 r14:0x7ffcf7b76a30 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.819253 global_time:3005, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:13688, ticks:7085587
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x2 rsi:0x7ffcf7b77a00 rdi:0xffffff9c rbp:0x2 rsp:0x681ffde0 r8:0x0 r9:0x7ffcf7b77727 r10:0x0 r11:0x246 r12:0x0 r13:0x0 r14:0x7ffcf7b77a00 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.821138 global_time:3006, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:13688, ticks:7085587
rax:0x12 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x2 rsi:0x7ffcf7b77a00 rdi:0xffffff9c rbp:0x2 rsp:0x681ffde0 r8:0x0 r9:0x7ffcf7b77727 r10:0x0 r11:0x246 r12:0x0 r13:0x0 r14:0x7ffcf7b77a00 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.821741 global_time:3007, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:13688, ticks:7085881
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x1100800000 rbp:0x9 rsp:0x681ffde0 r8:0x12 r9:0x0 r10:0x1 r11:0x246 r12:0x3 r13:0x1000 r14:0x1 r15:0x12 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.878697 global_time:3008, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:13688, ticks:7085881
rax:0x1100800000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x1100800000 rbp:0x9 rsp:0x681ffde0 r8:0x12 r9:0x0 r10:0x1 r11:0x246 r12:0x3 r13:0x1000 r14:0x1 r15:0x12 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7f6de56a4a80 gs_base:0x0
{ map_file:"/sys/devices/pci0000:00/0000:00:04.0/resource1", addr:0x1100800000, length:0x1000, prot_flags:"rw-s", file_offset:0x0, device:20, inode:12252, data_file:"", data_offset:0x0, file_size:0x1000 }
{ tid:13688, addr:0x1100800000, length:0x1000 }
{ tid:13688, addr:0x7f6de548d0da, length:0x1 }
}
{
real_time:1465.879452 global_time:3009, event:`SYSCALL: write' (state:ENTERING_SYSCALL) tid:13688, ticks:7086065
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x29 rsi:0x1f9d9a0 rdi:0x1 rbp:0x1f9d9a0 rsp:0x681ffde0 r8:0x0 r9:0x7f6de56a4a80 r10:0x0 r11:0x246 r12:0x29 r13:0x1 r14:0x0 r15:0x29 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.879895 global_time:3010, event:`SYSCALL: write' (state:EXITING_SYSCALL) tid:13688, ticks:7086065
rax:0x29 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x29 rsi:0x1f9d9a0 rdi:0x1 rbp:0x1f9d9a0 rsp:0x681ffde0 r8:0x0 r9:0x7f6de56a4a80 r10:0x0 r11:0x246 r12:0x29 r13:0x1 r14:0x0 r15:0x29 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.880553 global_time:3011, event:`SYSCALLBUF_FLUSH' tid:13688, ticks:7087157
{ syscall:'time', ret:0x5f3339b4, size:0x10 }
{ syscall:'getpid', ret:0x3578, size:0x10 }
{ syscall:'sendto', ret:0x52, size:0x10 }
}
{
real_time:1465.880603 global_time:3012, event:`SYSCALL: close' (state:ENTERING_SYSCALL) tid:13688, ticks:7087157
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1f9deb8 rdi:0x12 rbp:0x3 rsp:0x681ffde0 r8:0x2054b40 r9:0x7 r10:0x4000 r11:0x246 r12:0x0 r13:0x0 r14:0x0 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.880628 global_time:3013, event:`SYSCALLBUF_RESET' tid:13688, ticks:7087157
}
{
real_time:1465.880821 global_time:3014, event:`SYSCALL: close' (state:EXITING_SYSCALL) tid:13688, ticks:7087157
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1f9deb8 rdi:0x12 rbp:0x3 rsp:0x681ffde0 r8:0x2054b40 r9:0x7 r10:0x4000 r11:0x246 r12:0x0 r13:0x0 r14:0x0 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3 fs_base:0x7f6de56a4a80 gs_base:0x0
{ tid:13688, addr:0x7f6de548d0da, length:0x1 }
}
{
real_time:1465.881574 global_time:3015, event:`SYSCALLBUF_FLUSH' tid:13688, ticks:7087361
{ syscall:'pread64', ret:0x1, size:0x11 }
{ syscall:'pread64', ret:0x2, size:0x12 }
{ syscall:'pread64', ret:0x2, size:0x12 }
}
{
real_time:1465.881616 global_time:3016, event:`SYSCALL: iopl' (state:ENTERING_SYSCALL) tid:13688, ticks:7087361
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7f6dd639d6e8 rsi:0x0 rdi:0x3 rbp:0xac rsp:0x681ffde0 r8:0x40 r9:0x11 r10:0x42 r11:0x246 r12:0x400630 r13:0x7ffcf7b78f90 r14:0x0 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xac fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.881636 global_time:3017, event:`SYSCALLBUF_RESET' tid:13688, ticks:7087361
}
{
real_time:1465.881761 global_time:3018, event:`SYSCALL: iopl' (state:EXITING_SYSCALL) tid:13688, ticks:7087361
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7f6dd639d6e8 rsi:0x0 rdi:0x3 rbp:0xac rsp:0x681ffde0 r8:0x40 r9:0x11 r10:0x42 r11:0x246 r12:0x400630 r13:0x7ffcf7b78f90 r14:0x0 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xac fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.882492 global_time:3019, event:`SYSCALLBUF_FLUSH' tid:13688, ticks:7089342
{ syscall:'openat', ret:0x12, size:0x10 }
{ syscall:'readlinkat', ret:0x28, size:0x38 }
{ syscall:'fstat', ret:0x0, size:0xa0 }
{ syscall:'getdents64', ret:0x48, size:0x58 }
{ syscall:'close', ret:0x0, size:0x10 }
{ syscall:'openat', ret:0x12, size:0x10 }
{ syscall:'readlinkat', ret:0x40, size:0x50 }
{ syscall:'fstat', ret:0x0, size:0xa0 }
{ syscall:'read', ret:0x7, size:0x17 }
{ syscall:'close', ret:0x0, size:0x10 }
}
{
real_time:1465.882531 global_time:3020, event:`SYSCALL: write' (state:ENTERING_SYSCALL) tid:13688, ticks:7089342
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x24 rsi:0x1f9d9a0 rdi:0x1 rbp:0x1f9d9a0 rsp:0x681ffde0 r8:0x0 r9:0x7f6de56a4a80 r10:0x0 r11:0x246 r12:0x24 r13:0x1 r14:0x0 r15:0x24 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.882557 global_time:3021, event:`SYSCALLBUF_RESET' tid:13688, ticks:7089342
}
{
real_time:1465.882720 global_time:3022, event:`SYSCALL: write' (state:EXITING_SYSCALL) tid:13688, ticks:7089342
rax:0x24 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x24 rsi:0x1f9d9a0 rdi:0x1 rbp:0x1f9d9a0 rsp:0x681ffde0 r8:0x0 r9:0x7f6de56a4a80 r10:0x0 r11:0x246 r12:0x24 r13:0x1 r14:0x0 r15:0x24 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x7f6de56a4a80 gs_base:0x0
}
{
real_time:1465.885255 global_time:3023, event:`SYSCALLBUF_FLUSH' tid:13688, ticks:7098173
{ syscall:'time', ret:0x5f3339b4, size:0x10 }
{ syscall:'getpid', ret:0x3578, size:0x10 }
{ syscall:'sendto', ret:0x4d, size:0x10 }
{ syscall:'stat', ret:0x0, size:0xa0 }
{ syscall:'stat', ret:0xfffffffffffffffe, size:0xa0 }
}
{
real_time:1465.885309 global_time:3024, event:`SYSCALL: write' (state:ENTERING_SYSCALL) tid:13688, ticks:7098173
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x4c rsi:0x1f9d9a0 rdi:0x1 rbp:0x1f9d9a0 rsp:0x681ffde0 r8:0x0 r9:0x7f6de56a4a80 r10:0x0 r11:0x246 r12:0x4c r13:0x1 r14:0x0 r15:0x4c rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x7f6de56a4a80 gs_base:0x0
}
=== Start rr backtrace:
/root/dev/rr/obj/bin/rr(_ZN2rr13dump_rr_stackEv+0x33)[0x8b5ed7]
/root/dev/rr/obj/bin/rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x168)[0x722e6e]
/root/dev/rr/obj/bin/rr[0x7509e1]
/root/dev/rr/obj/bin/rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x5a)[0x750bce]
/root/dev/rr/obj/bin/rr(_ZN2rr13ReplaySession16flush_syscallbufEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x89c)[0x81e3d8]
/root/dev/rr/obj/bin/rr(_ZN2rr13ReplaySession18try_one_trace_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x170)[0x81eff8]
/root/dev/rr/obj/bin/rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x160)[0x820298]
/root/dev/rr/obj/bin/rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandEl+0xf1)[0x83a343]
/root/dev/rr/obj/bin/rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0x56e)[0x720546]
/root/dev/rr/obj/bin/rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x4cb)[0x722147]
/root/dev/rr/obj/bin/rr[0x816ce1]
/root/dev/rr/obj/bin/rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x3ba)[0x817620]
/root/dev/rr/obj/bin/rr(main+0x20c)[0x8cfa03]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f71552d26a3]
/root/dev/rr/obj/bin/rr(_start+0x2e)[0x69456e]
=== End rr backtrace
Raw
rr-with-dpdk_20200813T230000.log
# rr error on Packet's c3.small.x86
[root@ams1-c3 ~]# ./rr record ./dpdk-example --no-huge --log-level lib.eal:debug
rr: Saving execution to trace directory `/root/.local/share/rr/dpdk-example-1'.
[FATAL /home/ds/dev/rr/src/AddressSpace.cc:314:map_rr_page() errno: ENOENT]
(task 7028 (rec:7028) at time 14)
-> Assertion `page.is_open()' failed to hold. Failed to open rr_page file /root/../share/rr/rr_page_64
Tail of trace dump:
=== Start rr backtrace:
./rr(_ZN2rr13dump_rr_stackEv+0x33)[0x8b5ed7]
./rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x168)[0x722e6e]
./rr[0x7509e1]
./rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x5a)[0x750bce]
./rr(_ZN2rr12AddressSpace11map_rr_pageERNS_18AutoRemoteSyscallsE+0x119)[0x695cff]
./rr(_ZN2rr12AddressSpace17post_exec_syscallEPNS_4TaskE+0x107)[0x696e9f]
./rr(_ZN2rr4Task17post_exec_syscallEv+0x5f)[0x87c301]
./rr[0x7a5dbe]
./rr[0x7bec3d]
./rr[0x7aa7e0]
./rr(_ZN2rr19rec_process_syscallEPNS_10RecordTaskE+0xd2)[0x7aa922]
./rr(_ZN2rr13RecordSession21syscall_state_changedEPNS_10RecordTaskEPNS0_9StepStateE+0xc95)[0x78dd9d]
./rr(_ZN2rr13RecordSession11record_stepEv+0x3e7)[0x7931fb]
./rr[0x786336]
./rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x382)[0x786e50]
./rr(main+0x20c)[0x8cfa03]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7fd43f59c873]
./rr(_start+0x2e)[0x69456e]
=== End rr backtrace
Launch gdb with
gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:7028' /root/dpdk-example
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment