100Gbs Performance Optimization

100Gbs.md

100Gbs

• install latest sfutil
• sfupdate
• update sfc
• install nmon (optional for debug abd monitoring)
• install either iperf, iperf3, netperf, sockperf, sfnettest
• ifconfig p1p1 mtu 64, 1500, 9200
• NUMA taskset -c 0,1 (CPU Pinned)
• Open one socket per core.
• BIOS HT Disabled
• BIOS Performance Profile / Power
• systemctl stop irqbalance (IRQ pinning)
• sfc.conf => sfc module performance_profile=throughput
• echo 1000000 > /proc/sys/net/core/rmem_max
• ethtool –G p1p1 rx 512
• echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
• echo 512 | tee /sys/class/net/pp/queues/rx-*/rps_flow_cnt
• NOTE: still need to investigate MSI-Vector events.
• NOTE: still need to investigate iperf3.
• NOTE: traffic generator tx explanation.

Numa Locality Example:

1. For Interface p1p1:

ls -l /sys/class/net/p1p1/device
ethN -> /sys/class/net/p1p1/device -> 
../../../0000:83:00.0

2. Then cat numa_node:

/sys/devices/pci0000:80/0000:80:03.0/0000:83:00.0/numa_node
1

3. The ouput is the nearest numa node. All the cpu cores on this numa node are:

# cat /sys/devices/pci0000:80/0000:80:03.0/0000:83:00.0/local_cpulist
1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47

Intel Xeon Models:

• Server: Intel Xeon Gold 6240Y CPU @ 2.60GHz
• Generator: Intel Xeon Platinum 8180 CPU @ 2.50GHz
• Generator: Intel Xeon Platinum 8260M CPU @ 2.40GHz