Skip to content

Instantly share code, notes, and snippets.

@TJM
Last active November 2, 2020 19:16
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save TJM/31b68f12b2bc962141ce1eb27f7bcb10 to your computer and use it in GitHub Desktop.
Save TJM/31b68f12b2bc962141ce1eb27f7bcb10 to your computer and use it in GitHub Desktop.
Dealing with Orphaned pod messages (Orphaned pod found - but volume paths are still present on disk)
#!/bin/bash -eu
#
# This script is designed to be run as a cron job periodically to
# clean up the Orphaned Pods. Use at your own risk!
#
## Settings (can be passed as environment variables)
LOGFILE=${LOGFILE:-/var/log/messages} # what log file to process
KUBELET_PODS_DIR=${KUBELET_DIR:-/var/lib/kubelet/pods} # where to find pods to remove
DEBUG=${DEBUG:-0} # more debug output
## Script Starts Here
last_pod=""
# Retrieve the most recent "Orphaned pod" log entry (tac = backwards cat)
# NOTE: If there are no orphaned pods, then pod==last_pod, so its skips :)
while pod=$(tac ${LOGFILE} | grep -m 1 'Orphaned pod' | grep -o '".*"' | sed 's/"//g'); do
if [[ $pod != $last_pod ]]; then
last_pod=$pod
pod_dir="${KUBELET_PODS_DIR}/${pod}"
if [[ -d $pod_dir ]]; then
# echo "$pod exist"
rm -rf "${pod_dir}/volumes"
if [[ $? == 0 ]]; then
echo "Cleaned orphaned pod volumes: $pod"
else
echo "ERROR: Pod cannot be deleted: ${pod_dir}" >&2
fi
else
[[ $DEBUG == 1 ]] && echo "DEBUG: $pod doesn't exist in directory (${pod_dir})"
fi
else
[[ $DEBUG == 1 ]] && echo "DEBUG: No (more) orphaned pods found."
break
fi
sleep 1
done
@TJM
Copy link
Author

TJM commented Aug 13, 2020

I would like to see if we can make the "find most recent Orphaned pod" more efficient:

while pod=$(tac ${LOGFILE} | grep -m 1 'Orphaned pod' | grep -o '".*"' | sed 's/"//g'); do
  • tac = backwards cat (just found out about this, I usually scoff at people that use "cat file | grep" ;)
  • Double grep :(
  • sed to dequote

I feel like there is probably a better way to handle that... I am also not sure whether using tac LOGFILE | grep -m 1 'match' is more or less efficient than grep 'match' LOGFILE | tail -1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment