Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
k8s Network Trobleshooting

Network Troubleshooting

Inspecting Conntrack Connection Tracking

  • Prior to version 1.11, Kubernetes used iptables NAT and the conntrack kernel module to track connections. To list all the connections currently being tracked, use the conntrack command:
  • To list conntrack-tracked connections to a particular destination address, use the -d flag:
    conntrack -L -d

Node connection table full (issues making reliable connections to services)

  • It's possible your connection tracking table is full and new connections are being dropped. If that's the case you may see messages like the following in your system logs:

    $ tail -f /var/log/syslog

    error: Jul 12 15:32:11 worker-528 kernel: nf_conntrack: table full, dropping packet.

  • Check the maximum number of connections to tract.

    $ sysctl net.netfilter.nf_conntrack_max
  • To set a new value, use the -w flag:

    sysctl -w net.netfilter.nf_conntrack_max=198000
  • To make this setting permanent, add it to the sysctl.conf file:

    $ vi /etc/sysctl.conf
    . . .
    net.ipv4.netfilter.ip_conntrack_max = 198000

Inspecting Iptables Rules

  • Prior to version 1.11, Kubernetes used iptables NAT to implement virtual IP translation and load balancing for Service IPs.
  • To dump all iptables rules on a node, use the iptables-save command:
  • To list just the Kubernetes Service NAT rules, use the iptables command and the -L flag to specify the correct chain:
    iptables -t nat -L KUBE-SERVICES


  • Check ufw(Uncomplicated firewall) status
    $ sudo ufw status
    • if the status is
    Status: active
    • enable it
    $ sudo ufw enable
    • Check status to verify:
    $ sudo ufw status verbose
    Status: active
    Logging: on (low)
    Default: deny (incoming), allow (outgoing), disabled (routed)
    New profiles: skip

For further more debugging use iptables to debug related articles

For kubernetes: A reason for unexplained connection timeouts on Kubernetes/Docker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment