Skip to content

Instantly share code, notes, and snippets.

@zhangwei1984
Last active April 9, 2017 23:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save zhangwei1984/cedd4293d6b2cab6c21ddd0fc17a5cc6 to your computer and use it in GitHub Desktop.
Save zhangwei1984/cedd4293d6b2cab6c21ddd0fc17a5cc6 to your computer and use it in GitHub Desktop.
Two machines with 10G NIC directly connect each other
nn22 and nn23
step 1: create one VF on nn23
echo 1 > /sys/bus/pci/devices/0000:07:00.0/sriov_numvfs
Before bind VF to vfid-pci, try to ping this VF from nn22 (IP 12.0.0.22)
zhangwei1984@nimbnode23:~/bifurcation/dpdk-17.02$ sudo ./usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
<none>
Network devices using kernel driver
===================================
0000:05:00.0 '82576 Gigabit Network Connection' if=em1 drv=igb unused=igb_uio,vfio-pci *Active*
0000:05:00.1 '82576 Gigabit Network Connection' if=em2 drv=igb unused=igb_uio,vfio-pci
0000:07:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=p2p1 drv=ixgbe unused=igb_uio,vfio-pci
0000:07:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=p2p2 drv=ixgbe unused=igb_uio,vfio-pci
0000:07:10.0 '82599 Ethernet Controller Virtual Function' if=p2p1_0 drv=ixgbevf unused=igb_uio,vfio-pci *Active*
0000:07:10.1 '82599 Ethernet Controller Virtual Function' if=p2p2_0 drv=ixgbevf unused=igb_uio,vfio-pci
step 2: assign IP to this VF p2p1_0 12.0.0.23
p2p1 Link encap:Ethernet HWaddr 90:e2:ba:4a:e8:08
inet6 addr: fe80::92e2:baff:fe4a:e808/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1847 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:111124 (111.1 KB) TX bytes:0 (0.0 B)
p2p1_0 Link encap:Ethernet HWaddr 2a:6b:c5:99:c1:e7
inet addr:12.0.0.23 Bcast:12.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::286b:c5ff:fe99:c1e7/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1836 errors:0 dropped:0 overruns:0 frame:0
TX packets:1844 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:110160 (110.1 KB) TX bytes:77760 (77.7 KB)
From the tcpdump -i p2p1_0, it gave the reply.
tcpdump output for p2p1_0
19:32:26.495373 ARP, Request who-has 12.0.0.23 tell 12.0.0.22, length 46
19:32:26.495386 ARP, Reply 12.0.0.23 is-at 2a:6b:c5:99:c1:e7 (oui Unknown), length 28
19:32:27.493077 ARP, Request who-has 12.0.0.23 tell 12.0.0.22, length 46
19:32:27.493098 ARP, Reply 12.0.0.23 is-at 2a:6b:c5:99:c1:e7 (oui Unknown), length 28
Also ifconfig counter increases. It seems that the arp reply should be sent out.
But nn22 does not receive the packets. ifconfig drop counter is 0.
If using ethtool checks the counter, tx counter is 0.
NIC statistics:
rx_packets: 29
tx_packets: 0
rx_bytes: 1740
tx_bytes: 0
tx_busy: 0
tx_restart_queue: 0
tx_timeout_count: 1
multicast: 0
rx_csum_offload_errors: 0
rx_bp_poll_yield: 0
rx_bp_cleaned: 0
rx_bp_misses: 0
tx_bp_napi_yield: 0
tx_bp_cleaned: 0
tx_bp_misses: 0
why ifconfig and ethtool does not match each other. ifconfig shows tx increases.
but ethtool shows tx is 0. p2p1 tx is also 0 from ethtool stats.
Do you have any idea how to debug this problem?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment