How to identify and remove resources stuck in Terminating state

Kubernetes-Stuck-Resources.md

Namespaces

List all resources within a namespace

NS=some-namespace

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n $NS

Force-delete namespace with stuck finalizer (unsafe)

kubectl proxy &
kubectl get ns $NS -o json \
  | jq '.spec.finalizers=[]' \
  | curl -X PUT http://localhost:8001/api/v1/namespaces/$NS/finalize -H "Content-Type: application/json" --data @-

Pods

Failure to umount secret/configmap

# POD_UID=2dc52da6-50c7-4c39-af65-d99d548761d1
# mount |grep $POD_UID
tmpfs on /var/lib/kubelet/pods/2dc52da6-50c7-4c39-af65-d99d548761d1/volumes/kubernetes.io~secret/mypod-api-token-7qmcf type tmpfs (rw,relatime,seclabel)
# umount /var/lib/kubelet/pods/2dc52da6-50c7-4c39-af65-d99d548761d1/volumes/kubernetes.io~secret/mypod-api-token-7qmcf

Running kubectl log ... returns No space left on device even if all devices are <100% usage

## look for inotify watches and instances limits:

# sysctl -a 2>/dev/null | grep inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
user.max_inotify_instances = 128
user.max_inotify_watches = 8192

DotNet apps are huge consumers of inotify watches:

# sysctl -w fs.inotify.max_queued_events=1024 \
            fs.inotify.max_user_instances=524288 \
            fs.inotify.max_user_watches=524288 \
            user.max_inotify_instances=524288 \
            user.max_inotify_watches=524288