The most important TCP tuning areas since kernel 4.9 are:
- packet pacing
- dynamic TSO sizing
- TCP small queues
- BBR TCP congestion algorithm
- Gb = gigabit
- GB = gigabyte
- NIC = Network Interface Card e.g. eth0
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control
$ grep 'CONFIG_TCP_CONG_BBR' /boot/config-$(uname -r)
$ grep 'CONFIG_NET_SCH_FQ' /boot/config-$(uname -r)
$ egrep 'CONFIG_TCP_CONG_BBR|CONFIG_NET_SCH_FQ' /boot/config-$(uname -r)
ip link show | grep mtu
ping -s 8972 -M do -c 4 10.100.100.1
ethtool -g enp2s0f0
ethtool -i enp2s0f0
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 4194304
net.core.wmem_default = 1048576
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_congestion_control=htcp
ip link set eth0 mtu 9000
/etc/modprobe.conf (assuming tg3):
alias eth0 e1000
options tg3 RxDescriptors=4096,4096 TxDescriptors=4096,4096
# or
/etc/rc.local
ethtool -G ethN rx 4096 tx 4096
ethtool -g eth0
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_congestion_control=htcp
net.ipv4.tcp_mtu_probing=1
net.core.default_qdisc = fq
tc qdisc add dev $ETH root fq maxrate Ngbit
tc qdisc add dev $ETH root fq maxrate 2.5gbit
ip link set eth0 mtu 9000
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.ipv4.tcp_congestion_control=htcp
net.ipv4.tcp_mtu_probing=1
net.core.default_qdisc = fq
sysctl -w net.ipv4.tcp_congestion_control=htcp
tcp_mtu_probing = 1
- https://www3.cs.stonybrook.edu/~anshul/imc19_bbr.pdf
- https://github.com/google/bbr
- https://blog.apnic.net/2020/01/10/when-to-use-and-not-use-bbr/
Samba on MacOS is pretty terrible by default compared to Windows. It will not autrefresh on remote changes unlike Windows. There are a few changes on the server side that can make life better for MacOS clients. Very large >10,000 directory entries will still be a bit slow but <1000 directory entires should be pretty fast with these changes. Most of the changes are documented here: https://wiki.samba.org/index.php/Configure_Samba_to_Work_Better_with_Mac_OS_X
Check samba version. This is quite important. You can view the version changes at https://wiki.samba.org/index.php/Main_Page. For my Ubuntu server it looks like this:
$ samba --version
Version 4.11.6-Ubuntu
$uname -r
5.4.0-52-generic
Apple extensions require support for extended attributes(xattr) - defaults to yes in Samba 4.9+ Load the necessary modules; the actual load order is important and also enable AAPL extensions
[global]
min protocol = SMB2
ea support = yes
vfs objects = catia fruit streams_xattr
fruit:metadata = stream
fruit:model = MacSamba
fruit:veto_appledouble = no
fruit:posix_rename = yes
fruit:zero_file_id = yes
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes
unix charset = utf8
vfs objects = fruit
fruit:aapl = yes
fruit:encoding = native
fruit:locking = none
fruit:resource = file
fruit:time machine = no
fruit:posix_rename = yes
readdir_attr:aapl_rsize = yes
readdir_attr:aapl_finder_info = yes
readdir_attr:aapl_max_access = no
mdns name = mdns
MacOS benefits from increasing this value. The client can make less directory listing requests.
Check both the hard limit and the soft limit.
ulimit -Hn -Sn
or
ulimit -a
or
cat /proc/sys/fs/file-max
In the file /etc/security/limits.conf add the following line with a value that makes sense. This will change the value for all users on the system.
* - nofile 16385
* hard nofile 4096
Example soft limit for max opened files * is for all users, You can also specifiy specific users by username
* soft nofile 1024
fs.file-max=500000
sysctl -p