Skip to content

Instantly share code, notes, and snippets.

@adleong
Last active January 31, 2023 21:03
Show Gist options
  • Save adleong/0203b0864af2c29ddb821dd48f339f49 to your computer and use it in GitHub Desktop.
Save adleong/0203b0864af2c29ddb821dd48f339f49 to your computer and use it in GitHub Desktop.
Detect half-open connections in Kubernetes
#!/bin/bash
pods=$(kubectl get po -A -l linkerd.io/control-plane-ns -ojsonpath="{range .items[*]}{.metadata.name} {.metadata.namespace}{'\n'}{end}")
IFS=" "
while read name namespace; do
tcp=$(kubectl debug -n $namespace $name --image=cr.l5d.io/linkerd/debug:stable-2.12.0 -it -- cat /proc/net/tcp)
close_wait=$(echo $tcp | awk 'BEGIN {cnt=0} $4==08 {cnt++} END {print cnt}')
fin_wait_2=$(echo $tcp | awk 'BEGIN {cnt=0} $4==05 {cnt++} END {print cnt}')
if [ "$close_wait" -gt "0" -o "$fin_wait_2" -gt "0" ]; then
echo "$name.$namespace has $close_wait sockets in CLOSE_WAIT and $fin_wait_2 sockets in FIN_WAIT_2"
else
echo "$name.$namespace is okay"
fi
done <<< "$pods"
@rootik
Copy link

rootik commented Sep 8, 2022

exec failed: container_linux.go:380: starting container process caused: exec: "cat": executable file not found in $PATH: unknown

@adleong
Copy link
Author

adleong commented Sep 9, 2022

Thanks, @rootik! Good catch: the script stopped working since the cat utility is no longer included in the linkerd-proxy container.

I've updated the script to instead use a debug ephemeral container.

@rootik
Copy link

rootik commented Sep 9, 2022

👍

@dwilliams782
Copy link

error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource").

Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8-gke.202", GitCommit:"88deae00580af268497b9656f216cb092b630563", GitTreeState:"clean", BuildDate:"2022-06-03T03:27:52Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}

@adleong
Copy link
Author

adleong commented Sep 12, 2022

@dwilliams782 yes, this requires that ephemeral container support is enabled on the cluster. otherwise you will need to add the debug containers at inject time. see: https://linkerd.io/2.12/tasks/using-the-debug-container/

@dwilliams782
Copy link

Cool! NP - I haven't dug into that debug container use case but it sounds like it might help with my most recent issue - link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment