Network troubleshooting notes

old-notes.txt

Network troubleshooting
1. Check /etc/sysctl.conf
	Specifically:
		* net.core.netdev_max_backlog
		* net.core.rmem_max
		* net.core.rmem_default
	Example values for default:
		* net.core.netdev_max_backlog = 1000
		* net.core.rmem_max = 131071
		* net.core.rmem_default = 163480

2. In general, dropped received packages = filled NIC buffer. Ex:
	* NIC ring buffers getting full and unable to cope-up with bursts of traffic
	* CPU receiving NIC interrupts is very busy and unable to process
	* Check hardware (cables)
	* Bug in NIC driver

3. If using Linux ethernet bridge, check if ebtables is filled and needs to be flushed

4. If using Linux ethernet bridge, some recommendations:
	* (do not query iptables for package routing): echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
	* (do not do additional processing for multicast packages):
		- echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_querier
		- echo 0 > /sys/devices/virtual/net/br0/bridge/multicast_snooping

5. Update kernel (make sure it's not a known bug with drivers/distro/kernel)

6. Example commands:
	* (stats view of dropped/failed packets): 		netstat -s | grep -Ei 'drop|fail'
	* (view more stats): 					ethtool -S <NIC> (eth0 etc)
	* (view dropped packets with awk directly from /proc): 	awk '{ print $1,$5 }' /proc/net/dev 
	* (view debug messages for NIC): 			dmesg | grep e1000
	* (view link stats):					ip -s link show <NIC> (eth0)
	

7. Example values to check:
	* (Enable BPF jit):	/sbin/sysctl -w net.core.bpf_jit_enable=1
	* (Net buffer sizes):	/sbin/sysctl -w net.core.rmem_default=8388608
				/sbin/sysctl -w net.core.wmem_default=8388608

	* (UDP buffer sizes):	/sbin/sysctl -w net.ipv4.udp_rmem_min=131072
				/sbin/sysctl -w net.ipv4.udp_rmem_min=131072
				/sbin/sysctl -w net.ipv4.udp_wmem_min=131072
				/sbin/sysctl -w net.core.netdev_max_backlog=1000000
				/sbin/sysctl -w net.core.rmem_max=67108864

	* (TCP buffer sizes):	/sbin/sysctl -w net.ipv4.tcp_wmem='1048576 4194304 16777216'
				/sbin/sysctl -w net.ipv4.tcp_rmem='1048576 4194304 16777216'