Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
k8s Network Trobleshooting

Network Troubleshooting

Inspecting Conntrack Connection Tracking

  • Prior to version 1.11, Kubernetes used iptables NAT and the conntrack kernel module to track connections. To list all the connections currently being tracked, use the conntrack command:
  • To list conntrack-tracked connections to a particular destination address, use the -d flag:
    conntrack -L -d 10.32.0.1
    

Node connection table full (issues making reliable connections to services)

  • It's possible your connection tracking table is full and new connections are being dropped. If that's the case you may see messages like the following in your system logs:

    $ tail -f /var/log/syslog
    

    error: Jul 12 15:32:11 worker-528 kernel: nf_conntrack: table full, dropping packet.

  • Check the maximum number of connections to tract.

    $ sysctl net.netfilter.nf_conntrack_max
    
  • To set a new value, use the -w flag:

    sysctl -w net.netfilter.nf_conntrack_max=198000
    
  • To make this setting permanent, add it to the sysctl.conf file:

    $ vi /etc/sysctl.conf
    . . .
    net.ipv4.netfilter.ip_conntrack_max = 198000
    

Inspecting Iptables Rules

  • Prior to version 1.11, Kubernetes used iptables NAT to implement virtual IP translation and load balancing for Service IPs.
  • To dump all iptables rules on a node, use the iptables-save command:
    iptables-save
    
  • To list just the Kubernetes Service NAT rules, use the iptables command and the -L flag to specify the correct chain:
    iptables -t nat -L KUBE-SERVICES
    

USE NETFILTER ON YOUR LINUX SYSTEM: ENABLING A PACKET-FILTERING FIREWALL

  • Check ufw(Uncomplicated firewall) status
    $ sudo ufw status
    
    • if the status is
    Status: active
    
    • enable it
    $ sudo ufw enable
    
    • Check status to verify:
    $ sudo ufw status verbose
    
    Output:
    Status: active
    
    Logging: on (low)
    
    Default: deny (incoming), allow (outgoing), disabled (routed)
    
    New profiles: skip
    

For further more debugging use iptables to debug related articles https://www.dummies.com/computers/operating-systems/linux/how-to-use-netfilter-on-your-linux-system-enabling-a-packet-filtering-firewall/

For kubernetes: A reason for unexplained connection timeouts on Kubernetes/Docker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment