Detect half-open connections in Kubernetes

lost-sock.sh

#!/bin/bash

pods=$(kubectl get po -A -l linkerd.io/control-plane-ns -ojsonpath="{range .items[*]}{.metadata.name} {.metadata.namespace}{'\n'}{end}")

IFS=" "

while read name namespace; do
  tcp=$(kubectl debug -n $namespace $name --image=cr.l5d.io/linkerd/debug:stable-2.12.0 -it -- cat /proc/net/tcp)
  close_wait=$(echo $tcp | awk 'BEGIN {cnt=0} $4==08 {cnt++} END {print cnt}')
  fin_wait_2=$(echo $tcp | awk 'BEGIN {cnt=0} $4==05 {cnt++} END {print cnt}')
  if [ "$close_wait" -gt "0" -o "$fin_wait_2" -gt "0" ]; then
        echo "$name.$namespace has $close_wait sockets in CLOSE_WAIT and $fin_wait_2 sockets in FIN_WAIT_2"
  else
        echo "$name.$namespace is okay"
  fi
done <<< "$pods"

exec failed: container_linux.go:380: starting container process caused: exec: "cat": executable file not found in $PATH: unknown

Thanks, @rootik! Good catch: the script stopped working since the cat utility is no longer included in the linkerd-proxy container.

I've updated the script to instead use a debug ephemeral container.

👍

error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource").

Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8-gke.202", GitCommit:"88deae00580af268497b9656f216cb092b630563", GitTreeState:"clean", BuildDate:"2022-06-03T03:27:52Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}

@dwilliams782 yes, this requires that ephemeral container support is enabled on the cluster. otherwise you will need to add the debug containers at inject time. see: https://linkerd.io/2.12/tasks/using-the-debug-container/

Cool! NP - I haven't dug into that debug container use case but it sounds like it might help with my most recent issue - link