Skip to content

Instantly share code, notes, and snippets.

@mcastelino
Forked from krsna1729/canal.md
Created July 10, 2019 16:27
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mcastelino/fb841c7e7d26b08240e0a19d3b95c0a7 to your computer and use it in GitHub Desktop.
Save mcastelino/fb841c7e7d26b08240e0a19d3b95c0a7 to your computer and use it in GitHub Desktop.
tracing network stack for MACVLAN IPVLAN

MACVLAN

ip netns del net1
ip netns add net1
ip link add macvlan1 link enp130s0f0 type macvlan mode bridge
ip link set macvlan1 netns net1 up
ip netns exec net1 ip a add 192.168.1.200/24 dev macvlan1
ip netns exec net1 ping 192.168.1.101 -c1

Output

ip netns exec net1 perf trace --no-syscalls --event 'net:*' ping 192.168.1.101 -c1 > /dev/null

0.000 ping/5136 net:net_dev_queue:dev=macvlan1 skbaddr=0xffff9046524c9b00 len=98
0.008 ping/5136 net:net_dev_start_xmit:dev=macvlan1 queue_mapping=0 skbaddr=0xffff9046524c9b00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.011 ping/5136 net:net_dev_queue:dev=enp130s0f0 skbaddr=0xffff9046524c9b00 len=98
0.016 ping/5136 net:net_dev_start_xmit:dev=enp130s0f0 queue_mapping=52 skbaddr=0xffff9046524c9b00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.025 ping/5136 net:net_dev_xmit:dev=enp130s0f0 skbaddr=0xffff9046524c9b00 len=98 rc=0
0.026 ping/5136 net:net_dev_xmit:dev=macvlan1 skbaddr=0xffff9046524c9b00 len=98 rc=0
ip netns exec net1 trace-cmd record -e bridge -e qdisc -e fib -e tcp -e udp -e sock -e napi -e net -e skb -e fib6 ping 192.168.1.101 -c1 > /dev/null
trace-cmd report

ping-5924  [072] 710625.579565: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 0.0.0.0/53849 -> 192.168.1.101/1025 tos 0 scope 0 flags 0 ==> dev macvlan1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-5924  [072] 710625.579573: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 192.168.1.200/53849 -> 192.168.1.101/1025 tos 0 scope 0 flags 0 ==> dev macvlan1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-5924  [072] 710625.579607: fib_table_lookup:     table 254 oif 0 iif 1 proto 1 0.0.0.0/0 -> 192.168.1.101/0 tos 0 scope 0 flags 0 ==> dev macvlan1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-5924  [072] 710625.579613: net_dev_queue:        dev=macvlan1 skbaddr=0xffff9046524c9400 len=98
ping-5924  [072] 710625.579617: net_dev_start_xmit:   dev=macvlan1 queue_mapping=0 skbaddr=0xffff9046524c9400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
ping-5924  [072] 710625.579619: net_dev_queue:        dev=enp130s0f0 skbaddr=0xffff9046524c9400 len=98
ping-5924  [072] 710625.579624: qdisc_dequeue:        dequeue ifindex=6 qdisc handle=0x0 parent=0x49 txq_state=0x0 packets=1 skbaddr=0xffff9046524c9400
ping-5924  [072] 710625.579626: net_dev_start_xmit:   dev=enp130s0f0 queue_mapping=72 skbaddr=0xffff9046524c9400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
ping-5924  [072] 710625.579632: net_dev_xmit:         dev=enp130s0f0 skbaddr=0xffff9046524c9400 len=98 rc=0
ping-5924  [072] 710625.579635: net_dev_xmit:         dev=macvlan1 skbaddr=0xffff9046524c9400 len=98 rc=0
ping-5924  [072] 710625.579639: consume_skb:          skbaddr=0xffff9046524c9400
ping-5924  [072] 710625.579645: napi_poll:            napi poll on napi struct 0xffff904e52e82858 for device enp130s0f0 work 0 budget 64
ping-5924  [072] 710625.579648: qdisc_dequeue:        dequeue ifindex=6 qdisc handle=0x0 parent=0x49 txq_state=0x0 packets=0 skbaddr=(nil)
<idle>-0     [000] 710625.579724: napi_gro_receive_entry: dev=enp130s0f0 napi_id=0x161 queue_mapping=1 skbaddr=0xffff90462899bc00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=84 data_len=0 truesize=896 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0
<idle>-0     [000] 710625.579732: netif_receive_skb:    dev=enp130s0f0 skbaddr=0xffff90462899bc00 len=84
<idle>-0     [000] 710625.579755: fib_table_lookup:     table 254 oif 0 iif 28 proto 0 192.168.1.101/0 -> 192.168.1.200/0 tos 0 scope 0 flags 0 ==> dev macvlan1 gw 0.0.0.0 src 192.168.1.200 err 0
<idle>-0     [000] 710625.579758: fib_table_lookup:     table 254 oif 0 iif 1 proto 0 192.168.1.200/0 -> 192.168.1.101/0 tos 0 scope 0 flags 0 ==> dev macvlan1 gw 0.0.0.0 src 192.168.1.200 err 0
<idle>-0     [000] 710625.579764: kfree_skb:            skbaddr=0xffff90462899bc00 protocol=2048 location=0xffffffffbdc6126b
ping-5924  [072] 710625.579766: skb_copy_datagram_iovec: skbaddr=0xffff90462899b000 len=84
ping-5924  [072] 710625.579770: consume_skb:          skbaddr=0xffff90462899b000
<idle>-0     [000] 710625.579770: napi_poll:            napi poll on napi struct 0xffff904e563e4058 for device enp130s0f0 work 1 budget 64

IPVLAN L3

ip netns del net1
ip netns add net1
ip link add name ipvl1 link enp130s0f0 type ipvlan mode l3
ip link set dev ipvl1 netns net1 up
ip netns exec net1 ip a add 192.168.1.200/24 dev ipvl1
ip netns exec net1 ip route add default dev ipvl1
ip netns exec net1 ping 192.168.1.101 -c1

Output (inconsistent reply)

ip netns exec net1 perf trace --no-syscalls --event 'net:*' ping 192.168.1.101 -c1 > /dev/null

0.000 ping/5325 net:net_dev_queue:dev=ipvl1 skbaddr=0xffff90462961d400 len=98
0.008 ping/5325 net:net_dev_start_xmit:dev=ipvl1 queue_mapping=0 skbaddr=0xffff90462961d400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.016 ping/5325 net:net_dev_queue:dev=enp130s0f0 skbaddr=0xffff90462961d400 len=98
0.021 ping/5325 net:net_dev_start_xmit:dev=enp130s0f0 queue_mapping=52 skbaddr=0xffff90462961d400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.029 ping/5325 net:net_dev_xmit:dev=enp130s0f0 skbaddr=0xffff90462961d400 len=98 rc=0
0.031 ping/5325 net:net_dev_xmit:dev=ipvl1 skbaddr=0xffff90462961d400 len=98 rc=0
ip netns exec net1 trace-cmd record -e bridge -e qdisc -e fib -e tcp -e udp -e sock -e napi -e net -e skb -e fib6 ping 192.168.1.101 -c1 > /dev/null
trace-cmd report

ping-6117  [066] 710799.655465: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 0.0.0.0/56476 -> 192.168.1.101/1025 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-6117  [066] 710799.655472: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 192.168.1.200/56476 -> 192.168.1.101/1025 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-6117  [066] 710799.655506: fib_table_lookup:     table 254 oif 0 iif 1 proto 1 0.0.0.0/0 -> 192.168.1.101/0 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-6117  [066] 710799.655513: net_dev_queue:        dev=ipvl1 skbaddr=0xffff90464c7d2400 len=98
ping-6117  [066] 710799.655516: net_dev_start_xmit:   dev=ipvl1 queue_mapping=0 skbaddr=0xffff90464c7d2400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
ping-6117  [066] 710799.655519: fib_table_lookup:     table 254 oif 6 iif 1 proto 0 192.168.1.200/0 -> 192.168.1.101/0 tos 0 scope 0 flags 1 ==> dev enp130s0f0 gw 0.0.0.0 src 192.168.1.102 err 0
ping-6117  [066] 710799.655522: net_dev_queue:        dev=enp130s0f0 skbaddr=0xffff90464c7d2400 len=98
ping-6117  [066] 710799.655529: qdisc_dequeue:        dequeue ifindex=6 qdisc handle=0x0 parent=0x43 txq_state=0x0 packets=1 skbaddr=0xffff90464c7d2400
ping-6117  [066] 710799.655531: net_dev_start_xmit:   dev=enp130s0f0 queue_mapping=66 skbaddr=0xffff90464c7d2400 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
ping-6117  [066] 710799.655536: net_dev_xmit:         dev=enp130s0f0 skbaddr=0xffff90464c7d2400 len=98 rc=0
ping-6117  [066] 710799.655539: qdisc_dequeue:        dequeue ifindex=6 qdisc handle=0x0 parent=0x43 txq_state=0x0 packets=0 skbaddr=(nil)
ping-6117  [066] 710799.655541: net_dev_xmit:         dev=ipvl1 skbaddr=0xffff90464c7d2400 len=98 rc=0
ping-6117  [066] 710799.655544: consume_skb:          skbaddr=0xffff90464c7d2400
ping-6117  [066] 710799.655549: napi_poll:            napi poll on napi struct 0xffff904e52e84c58 for device enp130s0f0 work 0 budget 64

......

IPVLAN L2

ip netns del net1
ip netns add net1
ip link add name ipvl1 link enp130s0f0 type ipvlan mode l2
ip link set dev ipvl1 netns net1 up
ip netns exec net1 ip a add 192.168.1.200/24 dev ipvl1
ip netns exec net1 ping 192.168.1.101 -c1
ip netns exec net1 perf trace --no-syscalls --event 'net:*' ping 192.168.1.101 -c1 > /dev/null

Output

0.000 ping/5182 net:net_dev_queue:dev=ipvl1 skbaddr=0xffff9045f6ea7500 len=98
0.009 ping/5182 net:net_dev_start_xmit:dev=ipvl1 queue_mapping=0 skbaddr=0xffff9045f6ea7500 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.012 ping/5182 net:net_dev_queue:dev=enp130s0f0 skbaddr=0xffff9045f6ea7500 len=98
0.034 ping/5182 net:net_dev_start_xmit:dev=enp130s0f0 queue_mapping=52 skbaddr=0xffff9045f6ea7500 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.042 ping/5182 net:net_dev_xmit:dev=enp130s0f0 skbaddr=0xffff9045f6ea7500 len=98 rc=0
0.045 ping/5182 net:net_dev_xmit:dev=ipvl1 skbaddr=0xffff9045f6ea7500 len=98 rc=0
ip netns exec net1 trace-cmd record -e bridge -e qdisc -e fib -e tcp -e udp -e sock -e napi -e net -e skb -e fib6 ping 192.168.1.101 -c1 > /dev/null
trace-cmd report

ping-6223  [064] 711150.615688: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 0.0.0.0/46426 -> 192.168.1.101/1025 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-6223  [064] 711150.615695: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 192.168.1.200/46426 -> 192.168.1.101/1025 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-6223  [064] 711150.615729: fib_table_lookup:     table 254 oif 0 iif 1 proto 1 0.0.0.0/0 -> 192.168.1.101/0 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
ping-6223  [064] 711150.615736: net_dev_queue:        dev=ipvl1 skbaddr=0xffff90464b5b0c00 len=98
ping-6223  [064] 711150.615740: net_dev_start_xmit:   dev=ipvl1 queue_mapping=0 skbaddr=0xffff90464b5b0c00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
ping-6223  [064] 711150.615742: net_dev_queue:        dev=enp130s0f0 skbaddr=0xffff90464b5b0c00 len=98
ping-6223  [064] 711150.615746: qdisc_dequeue:        dequeue ifindex=6 qdisc handle=0x0 parent=0x41 txq_state=0x0 packets=1 skbaddr=0xffff90464b5b0c00
ping-6223  [064] 711150.615748: net_dev_start_xmit:   dev=enp130s0f0 queue_mapping=64 skbaddr=0xffff90464b5b0c00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
ping-6223  [064] 711150.615753: net_dev_xmit:         dev=enp130s0f0 skbaddr=0xffff90464b5b0c00 len=98 rc=0
ping-6223  [064] 711150.615757: qdisc_dequeue:        dequeue ifindex=6 qdisc handle=0x0 parent=0x41 txq_state=0x0 packets=0 skbaddr=(nil)
ping-6223  [064] 711150.615759: net_dev_xmit:         dev=ipvl1 skbaddr=0xffff90464b5b0c00 len=98 rc=0
ping-6223  [064] 711150.615762: consume_skb:          skbaddr=0xffff90464b5b0c00
ping-6223  [064] 711150.615767: napi_poll:            napi poll on napi struct 0xffff904e52e84458 for device enp130s0f0 work 0 budget 64
<idle>-0     [000] 711150.615821: napi_gro_receive_entry: dev=enp130s0f0 napi_id=0x161 queue_mapping=1 skbaddr=0xffff9045f68bc100 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=84 data_len=0 truesize=896 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0
<idle>-0     [000] 711150.615829: netif_receive_skb:    dev=enp130s0f0 skbaddr=0xffff9045f68bc100 len=84
<idle>-0     [000] 711150.615852: fib_table_lookup:     table 254 oif 0 iif 30 proto 0 192.168.1.101/0 -> 192.168.1.200/0 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
<idle>-0     [000] 711150.615854: fib_table_lookup:     table 254 oif 0 iif 1 proto 0 192.168.1.200/0 -> 192.168.1.101/0 tos 0 scope 0 flags 0 ==> dev ipvl1 gw 0.0.0.0 src 192.168.1.200 err 0
<idle>-0     [000] 711150.615860: kfree_skb:            skbaddr=0xffff9045f68bc100 protocol=2048 location=0xffffffffbdc6126b
ping-6223  [064] 711150.615863: skb_copy_datagram_iovec: skbaddr=0xffff90462899b800 len=84
ping-6223  [064] 711150.615866: consume_skb:          skbaddr=0xffff90462899b800
<idle>-0     [000] 711150.615867: napi_poll:            napi poll on napi struct 0xffff904e563e4058 for device enp130s0f0 work 1 budget 64

Anamoly

0.000 ping/5095 net:net_dev_queue:dev=ipvl1 skbaddr=0xffff904613f11b00 len=42
0.008 ping/5095 net:net_dev_start_xmit:dev=ipvl1 queue_mapping=0 skbaddr=0xffff904613f11b00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0806 ip_summed=0 len=42 data_len=0 network_offset=14 transport_offset_valid=0 transport_offset=65533 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.012 ping/5095 net:net_dev_xmit:dev=ipvl1 skbaddr=0xffff904613f11b00 len=42 rc=0

Appendix

perf list 'net:*'

List of pre-defined events (to be used in -e):

  net:napi_gro_frags_entry                           [Tracepoint event]
  net:napi_gro_receive_entry                         [Tracepoint event]
  net:net_dev_queue                                  [Tracepoint event]
  net:net_dev_start_xmit                             [Tracepoint event]
  net:net_dev_xmit                                   [Tracepoint event]
  net:netif_receive_skb                              [Tracepoint event]
  net:netif_receive_skb_entry                        [Tracepoint event]
  net:netif_receive_skb_list_entry                   [Tracepoint event]
  net:netif_rx                                       [Tracepoint event]
  net:netif_rx_entry                                 [Tracepoint event]
  net:netif_rx_ni_entry                              [Tracepoint event]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment