Skip to content

Instantly share code, notes, and snippets.

View spencergilbert's full-sized avatar

Spencer Gilbert spencergilbert

  • Washington, DC
  • 04:32 (UTC -04:00)
View GitHub Profile

HOWTO: Clean Up Nomad Data Directory

Issue

When attempting to remove all of the data in the Nomad data directory, several directories and files are unable to be deleted. Many messages are logged to the console like:

rm: cannot remove ‘alloc/736f61b9-d7dc-cb73-0dd1-76b1b2ba032d/nomad-ui/secrets’: Device or resource busy
rm: cannot remove ‘alloc/ddcf5a78-5497-f4a4-a101-221fc4e0180b/fabio/alloc’: Device or resource busy
rm: cannot remove ‘alloc/ddcf5a78-5497-f4a4-a101-221fc4e0180b/fabio/secrets’: Device or resource busy
@patrick0057
patrick0057 / README.md
Last active October 6, 2020 13:23
Restoring RKE cluster with incorrect or missing rkestate file

Overview

When using RKE 0.2.0 and newer, if you have restored a cluster with the incorrect rkestate file you will end up a state where your infrastructure pods will not start. This includes all pods in kube-system, cattle-system and ingress-nginx. As a result of these core pods not starting, all of your workload pods will be unable to function correctly. If you find yourself in this situation you can use the directions below to fix the cluster.

Recovery

  1. Delete all service-account-token secrets in kube-system, cattle-system and ingress-nginx namespaces.
{
kubectl get secret -n cattle-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n cattle-system delete secret " $1) }'
kubectl get secret -n kube-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n kube-system delete secret " $1) }'