Skip to content

Instantly share code, notes, and snippets.

@rzezeski

rzezeski/poll.md

Created Sep 16, 2020
Embed
What would you like to do?
poll issue?

I hit this while running iperf3 across two VNICs on different NICs (no mac-loopback).

rpz@thunderhead:~$ iperf3 -c 192.168.99.101 -B 192.168.99.100 -t 10
Connecting to host 192.168.99.101, port 5201
[  4] local 192.168.99.100 port 48040 connected to 192.168.99.101 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  52.4 MBytes   440 Mbits/sec                  
[  4]   1.00-2.00   sec  56.6 MBytes   475 Mbits/sec                  
[  4]   2.00-3.00   sec  56.6 MBytes   475 Mbits/sec                  
[  4]   3.00-4.00   sec  56.6 MBytes   475 Mbits/sec                  
[  4]   4.00-5.00   sec  56.6 MBytes   475 Mbits/sec                  
[  4]   5.00-6.00   sec  56.6 MBytes   475 Mbits/sec                  
[  4]   6.00-7.00   sec  53.5 MBytes   449 Mbits/sec                  
[  4]   7.00-8.00   sec  56.6 MBytes   475 Mbits/sec                  
[  4]   8.00-9.00   sec  56.6 MBytes   475 Mbits/sec                  

iperf3: error - unable to receive control message: Connection timed out

The sender was hung on the 8-9s line for a long time. Here's the ps and pstack.

rpz@thunderhead:~$ ps -l -p 101138
 F S    UID    PID   PPID   C PRI NI     ADDR     SZ    WCHAN TTY         TIME CMD
 0 O    101 101138 101128   3  99 20        ?    729          pts/3       3:26 iperf3

rpz@thunderhead:~$ pstack 101138
101138: iperf3 -c 192.168.99.101 -B 192.168.99.100 -t 10
 fe86f775 __pollsys (80363c0, 2, 8036478, 0, 3, 40) + 15
 fe7fc102 pselect  (5, 80364e0, 80384e0, fe8da280, 8036478, 0) + 272
 fe7fc498 select   (5, 80364e0, 80384e0, 0, feeee214) + 89
 feed3369 iperf_run_client (8594008, 8598200, feecb34b, 8594008) + 109
 08051663 run      (8594008, 7, 803a5b0, 805172e) + f3
 08051758 main     (803a54c, fe8dd628, 803a588, 8051477, 7, 803a5b0) + 48
 08051477 _start_crt (7, 803a5b0, f5ab2681, 0, 0, 0) + 96
 0805134a _start   (7, 803a6a0, 803a6a7, 803a6aa, 803a6b9, 803a6bc) + 1a

Here is some mdb before it went away.

> 0t101138::pid2proc |::ps -t
S PID    PPID   PGID   SID    UID    FLAGS      ADDR             NAME
R 101138 101128 101138 101128 101    0x4a004000 fffffe5afaee7038 iperf3
        T  0xfffffe59ea5947e0 <TS_ONPROC>
> 
> 0xfffffe59ea5947e0::findstack
stack pointer for thread fffffe59ea5947e0 (iperf3/1): fffffe007dc18940
  fffffe007dc18980 apix_intr_exit+0x24()
  fffffe007dc189e0 apix_hilevel_intr_epilog+0xf1()
  fffffe007dc18a50 apix_do_interrupt+0x4c0()
  fffffe007dc18a60 _sys_rtt_ints_disabled+8()
  fffffe007dc18b60 mutex_owned+0x2d()
  fffffe007dc18bd0 socket_vop_poll+0x71()
  fffffe5afaee7038 4()
  fffffe5a2e0d8700 0xfffffe5a231eb618()
> 
> 0xfffffe59ea5947e0::findstack
stack pointer for thread fffffe59ea5947e0 (iperf3/1): fffffe007dc18be0
  fffffe007dc18c40 0xfffffe5afaee9126()
  fffffe007dc18d30 pcache_poll+0x508()
  fffffe007dc18e00 poll_common+0x49c()
  fffffe007dc18eb0 pollsys+0xfb()
  fffffe007dc18f00 _sys_sysenter_post_swapgs+0x24f()
> 0xfffffe59ea5947e0::findstack
stack pointer for thread fffffe59ea5947e0 (iperf3/1): fffffe007dc18910
  fffffe007dc18950 apix_intr_exit+0x24()
  fffffe007dc189b0 apix_hilevel_intr_epilog+0xf1()
  fffffe007dc18a20 apix_do_interrupt+0x4c0()
  fffffe007dc18a30 _sys_rtt_ints_disabled+8()
  fffffe007dc18b60 so_poll()
  fffffe007dc18bd0 socket_vop_poll+0x71()
  fffffe5afaee7038 4()
  fffffe5a2e0d8700 0xfffffe5a231eb618()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.