gistfile1.txt

# Experiument setup

Two machines with Intel 6700K Skylake at 4GHz, 32GB of DDR4 at 3GHz and Intel X710 quad port 10G NIC. 
Running Linux Kernel 4.4.3 with 8 * 1G hugepages
Both hosts are interconnected with two SFP+ cables on port 0 and 1.
Both experiments are executed after a clean reboot

# Experiment 1: uio_pci_generic
## Forwarding Machine

```bash
sudo modprobe uio_pci_generic
sudo ~/dpdk-2.2.0/tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.0
sudo ~/dpdk-2.2.0/tools/dpdk_nic_bind.py --bind=uio_pci_generic 0000:01:00.1
cd dpdk-2.2.0/examples/l2fwd
sudo ./build/l2fwd -c 0xff -n 4 -- -p 3

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000000 bytes
EAL: Virtual area found at 0x7f4b40000000 (size = 0x200000000)
EAL: Requesting 8 pages of size 1024MB from socket 0
EAL: TSC frequency is ~4007999 KHz
EAL: Master lcore 0 is ready (tid=d00658c0;cpuset=[0])
EAL: lcore 6 is ready (tid=cc14d700;cpuset=[6])
EAL: lcore 4 is ready (tid=cd14f700;cpuset=[4])
EAL: lcore 5 is ready (tid=cc94e700;cpuset=[5])
EAL: lcore 1 is ready (tid=ce952700;cpuset=[1])
EAL: lcore 3 is ready (tid=cd950700;cpuset=[3])
EAL: lcore 7 is ready (tid=cb94c700;cpuset=[7])
EAL: lcore 2 is ready (tid=ce151700;cpuset=[2])
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   PCI memory mapped at 0x7f4d40000000
EAL:   PCI memory mapped at 0x7f4d40800000
PMD: eth_i40e_dev_init(): FW 4.22 API 1.2 NVM 04.02.06 eetrack 8000152d
PMD: i40e_dcb_init_configure(): Failed to stop lldp
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
PMD: eth_i40e_dev_init(): Failed to init dcb.
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   PCI memory mapped at 0x7f4d40808000
EAL:   PCI memory mapped at 0x7f4d41008000
PMD: eth_i40e_dev_init(): FW 4.22 API 1.2 NVM 04.02.06 eetrack 8000152d
PMD: i40e_dcb_init_configure(): Failed to stop lldp
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
PMD: eth_i40e_dev_init(): Failed to init dcb.
EAL: PCI device 0000:01:00.2 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.3 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   probe driver: 8086:105e rte_em_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:00.1 on NUMA socket -1
EAL:   probe driver: 8086:105e rte_em_pmd
EAL:   Not managed by a supported kernel driver, skipped
Lcore 0: RX port 0
Lcore 1: RX port 1
Initializing port 0... PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
PMD: i40e_set_tx_function_flag(): Vector tx can be enabled on this txq.
PMD: i40e_set_tx_function(): Simple tx finally be used.
PMD: i40e_set_rx_function(): Port[0] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0.
done:
Port 0, MAC address: 68:05:CA:36:F1:E8

Initializing port 1... PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
PMD: i40e_set_tx_function_flag(): Vector tx can be enabled on this txq.
PMD: i40e_set_tx_function(): Simple tx finally be used.
PMD: i40e_set_rx_function(): Port[1] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1.
done:
Port 1, MAC address: 68:05:CA:36:F1:E9


Checking link statusdone
Port 0 Link Up - speed 10000 Mbps - full-duplex
Port 1 Link Up - speed 10000 Mbps - full-duplex
L2FWD: entering main loop on lcore 1
L2FWD:  -- lcoreid=1 portid=1
L2FWD: lcore 2 has nothing to do
L2FWD: lcore 3 has nothing to do
L2FWD: lcore 4 has nothing to do
L2FWD: lcore 5 has nothing to do
L2FWD: lcore 6 has nothing to do
L2FWD: entering main loop on lcore 0
L2FWD:  -- lcoreid=0 portid=0
```

## Traffic Generator
```bash
sudo ./build/MoonGen rfc2544/master.lua --txport 0 --rxport 1
[INFO]  Initializing DPDK. This will take a few seconds...
EAL: No free hugepages reported in hugepages-1048576kB
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
[INFO]  Found 2 usable devices:
   Device 0: 68:05:CA:36:F8:A0 (Intel Corporation Ethernet 10G 2P X710 Adapter)
   Device 1: 68:05:CA:36:F8:A1 (Intel Corporation Ethernet 10G 2P X710 Adapter)
[WARN]  You are running Linux >= 3.14, DDIO might not be working with DPDK in this setup!
[WARN]  This can cause a huge performance impact (one memory access per packet!) preventing MoonGen from reaching line rate.
[WARN]  Try using an older kernel (we recommend 3.13) if you see a low performance or huge cache miss ratio.
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
[INFO]  Waiting for devices to come up...
[INFO]  Device 0 (68:05:CA:36:F8:A0) is up: full-duplex 10000 MBit/s
[INFO]  Device 1 (68:05:CA:36:F8:A1) is up: full-duplex 10000 MBit/s
[INFO]  2 devices are up.
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
frame size(byte),duration(s),max loss rate(%),rate threshold(packets),rate(mpps) iter1,spkts(byte) iter1,rpkts(byte) iter1
configure: add to interface nil ip 198.18.1.1/24continue (y/n)? y
configure: add to interface nil ip 198.19.1.1/24continue (y/n)? y
starting iteration 1 for frameSize 64
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 10000 instead
unable to load snmp module
unable to load snmp module
unable to load mikrotik ssh module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
unable to load linux ssh module
sent 148754151 packets, received 18688096
rate 10000.000000 and packetloss 0.874369 => 0
changing rate from 10000 MBit/s to 5000 MBit/s
set queue 1 to rate 3809
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 3809 instead
unable to load snmp module
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
sent 81035703 packets, received 17106682
rate 5000.000000 and packetloss 0.788899 => 0
changing rate from 5000 MBit/s to 2500 MBit/s
set queue 1 to rate 1904
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 1904 instead
unable to load snmp module
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
sent 40522671 packets, received 16095232
rate 2500.000000 and packetloss 0.602809 => 0
changing rate from 2500 MBit/s to 1250 MBit/s
set queue 1 to rate 952
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 952 instead
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
sent 20264706 packets, received 15155945
rate 1250.000000 and packetloss 0.252101 => 0
changing rate from 1250 MBit/s to 625 MBit/s
set queue 1 to rate 476
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 476 instead
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load snmp module
unable to load linux ssh module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
sent 10665648 packets, received 10655494
rate 625.000000 and packetloss 0.000952 => 1
changing rate from 625 MBit/s to 938 MBit/s
set queue 1 to rate 714
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 714 instead
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
sent 14932008 packets, received 12706312
rate 938.000000 and packetloss 0.149055 => 0
changing rate from 938 MBit/s to 782 MBit/s
set queue 1 to rate 595
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 595 instead
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
sent 12798828 packets, received 11755353
rate 782.000000 and packetloss 0.081529 => 0
maximal rate for packetsize 64: 1.07 Mpps, 546.08 MBit/s, 716.73 MBit/s wire rate
```

# Experiment 2: uio, igb_uio

## Forwarding Machine

```bash
sudo modprobe uio
sudo insmod /home/simonj/dpdk-2.2.0/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
sudo ~/dpdk-2.2.0/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0
sudo ~/dpdk-2.2.0/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.1
cd dpdk-2.2.0/examples/l2fwd

sudo ./build/l2fwd -c 0xff -n 4 -- -p 3

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 2 on socket 0
EAL: Detected lcore 7 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000000 bytes
EAL: Virtual area found at 0x7fefc0000000 (size = 0x200000000)
EAL: Requesting 8 pages of size 1024MB from socket 0
EAL: TSC frequency is ~4008000 KHz
EAL: Master lcore 0 is ready (tid=63a698c0;cpuset=[0])
EAL: lcore 4 is ready (tid=60b53700;cpuset=[4])
EAL: lcore 2 is ready (tid=61b55700;cpuset=[2])
EAL: lcore 6 is ready (tid=5fb51700;cpuset=[6])
EAL: lcore 5 is ready (tid=60352700;cpuset=[5])
EAL: lcore 7 is ready (tid=5f350700;cpuset=[7])
EAL: lcore 1 is ready (tid=62356700;cpuset=[1])
EAL: lcore 3 is ready (tid=61354700;cpuset=[3])
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   PCI memory mapped at 0x7ff1c0000000
EAL:   PCI memory mapped at 0x7ff1c0800000
PMD: eth_i40e_dev_init(): FW 4.22 API 1.2 NVM 04.02.06 eetrack 8000152d
PMD: i40e_dcb_init_configure(): Failed to stop lldp
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
PMD: eth_i40e_dev_init(): Failed to init dcb.
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   PCI memory mapped at 0x7ff1c0808000
EAL:   PCI memory mapped at 0x7ff1c1008000
PMD: eth_i40e_dev_init(): FW 4.22 API 1.2 NVM 04.02.06 eetrack 8000152d
PMD: i40e_dcb_init_configure(): Failed to stop lldp
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
PMD: eth_i40e_dev_init(): Failed to init dcb.
EAL: PCI device 0000:01:00.2 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.3 on NUMA socket -1
EAL:   probe driver: 8086:1572 rte_i40e_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   probe driver: 8086:105e rte_em_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:04:00.1 on NUMA socket -1
EAL:   probe driver: 8086:105e rte_em_pmd
EAL:   Not managed by a supported kernel driver, skipped
Lcore 0: RX port 0
Lcore 1: RX port 1
Initializing port 0... PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
PMD: i40e_set_tx_function_flag(): Vector tx can be enabled on this txq.
PMD: i40e_set_tx_function(): Simple tx finally be used.
PMD: i40e_set_rx_function(): Port[0] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0.
done:
Port 0, MAC address: 68:05:CA:36:F1:E8

Initializing port 1... PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
PMD: i40e_set_tx_function_flag(): Vector tx can be enabled on this txq.
PMD: i40e_set_tx_function(): Simple tx finally be used.
PMD: i40e_set_rx_function(): Port[1] doesn't meet Vector Rx preconditions
PMD: i40e_set_rx_function(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1.
done:
Port 1, MAC address: 68:05:CA:36:F1:E9


Checking link statusdone
Port 0 Link Up - speed 10000 Mbps - full-duplex
Port 1 Link Up - speed 10000 Mbps - full-duplex
L2FWD: entering main loop on lcore 1
L2FWD:  -- lcoreid=1 portid=1
L2FWD: lcore 2 has nothing to do
L2FWD: lcore 3 has nothing to do
L2FWD: lcore 4 has nothing to do
L2FWD: lcore 5 has nothing to do
L2FWD: lcore 6 has nothing to do
L2FWD: lcore 7 has nothing to do
L2FWD: entering main loop on lcore 0
L2FWD:  -- lcoreid=0 portid=0
```

## Traffic Generator
```bash
sudo ./build/MoonGen rfc2544/master.lua --txport 0 --rxport 1
[INFO]  Initializing DPDK. This will take a few seconds...
EAL: No free hugepages reported in hugepages-1048576kB
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
PMD: i40e_dcb_init_configure(): DCBX configuration failed, err = 0, aq_err = 0.
[INFO]  Found 2 usable devices:
   Device 0: 68:05:CA:36:F8:A0 (Intel Corporation Ethernet 10G 2P X710 Adapter)
   Device 1: 68:05:CA:36:F8:A1 (Intel Corporation Ethernet 10G 2P X710 Adapter)
[WARN]  You are running Linux >= 3.14, DDIO might not be working with DPDK in this setup!
[WARN]  This can cause a huge performance impact (one memory access per packet!) preventing MoonGen from reaching line rate.
[WARN]  Try using an older kernel (we recommend 3.13) if you see a low performance or huge cache miss ratio.
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
[INFO]  Waiting for devices to come up...
[INFO]  Device 0 (68:05:CA:36:F8:A0) is up: full-duplex 10000 MBit/s
[INFO]  Device 1 (68:05:CA:36:F8:A1) is up: full-duplex 10000 MBit/s
[INFO]  2 devices are up.
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
frame size(byte),duration(s),max loss rate(%),rate threshold(packets),rate(mpps) iter1,spkts(byte) iter1,rpkts(byte) iter1
configure: add to interface nil ip 198.18.1.1/24continue (y/n)? y
configure: add to interface nil ip 198.19.1.1/24continue (y/n)? y
starting iteration 1 for frameSize 64
[WARN]  Per-queue rate limit NYI on this device, setting per-device rate limit to 10000 instead
unable to load snmp module
unable to load snmp module
unable to load mikrotik ssh module
unable to load freeBSD ssh module
unable to load mikrotik ssh module
unable to load linux ssh module
unable to load freeBSD ssh module
unable to load linux ssh module
sent 148754970 packets, received 148754029
rate 10000.000000 and packetloss 0.000006 => 1
maximal rate for packetsize 64: 14.88 Mpps, 7616.25 MBit/s, 9996.33 MBit/s wire rate
```