Skip to content

Instantly share code, notes, and snippets.

@JeanMertz
Created January 19, 2017 08:18
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save JeanMertz/c5f26e69f14910e828f85185987692b2 to your computer and use it in GitHub Desktop.
Save JeanMertz/c5f26e69f14910e828f85185987692b2 to your computer and use it in GitHub Desktop.
kube-dns
$ kubectl get po | grep kube-dns
kube-dns-2992416901-8nvjb 3/4 CrashLoopBackOff 101 1d
kube-dns-2992416901-n1jr6 4/4 Running 0 4d
kube-dns-autoscaler-2093529816-1nhwb 1/1 Running 2 5d
$ kubectl describe po kube-dns-2992416901-8nvjb | tail -n5
8h 3m 96 {kubelet gke-blendle-default-preemptible-pool--29506a72-k1hz} spec.containers{dnsmasq} NormaCreated (events with common reason combined)
8h 3m 96 {kubelet gke-blendle-default-preemptible-pool--29506a72-k1hz} spec.containers{dnsmasq} NormaStarted (events with common reason combined)
8h 12s 2289 {kubelet gke-blendle-default-preemptible-pool--29506a72-k1hz} spec.containers{dnsmasq} Warning BackOff Back-off restarting failed docker container
8h 12s 2259 {kubelet gke-blendle-default-preemptible-pool--29506a72-k1hz} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-2992416901-8nvjb_kube-system(02b2bdc4-dccc-11e6-808e-42010a84012a)"
$ kubectl logs kube-dns-2992416901-8nvjb dnsmasq
dnsmasq: failed to create inotify: No file descriptors available
$ kubectl get po -ojson kube-dns-2992416901-8nvjb | jq .spec.nodeName
"gke-blendle-default-preemptible-pool--29506a72-k1hz"
$ gcloud compute ssh gke-blendle-default-preemptible-pool--29506a72-k1hz -- -l core sudo df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 79360 12706 66654 17% /
devtmpfs 15448042 507 15447535 1% /dev
tmp 15448278 17 15448261 1% /tmp
run 15448278 3647 15444631 1% /run
shmfs 15448278 1 15448277 1% /dev/shm
/dev/sda1 12786720 1423094 11363626 12% /var
/dev/sda8 4096 11 4085 1% /usr/share/oem
media 15448278 1 15448277 1% /media
tmpfs 15448278 13 15448265 1% /sys/fs/cgroup
tmpfs 15448278 1 15448277 1% /mnt/disks
tmpfs 15448278 45 15448233 1% /var/lib/cloud
overlayfs 15448278 60 15448218 1% /etc
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/09be0776-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-lylm2
tmpfs 15448278 17 15448261 1% /var/lib/kubelet/pods/9ae5ddaa-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/8e944e80-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/de2a7882-dab1-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-ktnvk
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/4be0a0ce-dbfa-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-ktnvk
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/051ee6f8-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-zsa3d
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/83a7be69-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/e02a0bf2-dab1-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-xnayj
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/a393e6c6-dd96-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/343452ad-dd96-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/d5bd44e7-dbf9-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-ktnvk
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/94fc8bd8-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-7e03u
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/99567db4-dc0f-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/7cd51a44-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/02b2bdc4-dccc-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-q0d1q
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/70e9e7fd-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-tsbym
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/cd8f5b35-dd95-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/713bf292-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/962fd5f1-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-rphkp
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/1bf20af4-dd95-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/e42096b3-dab1-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-7ipxx
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/c7a6b581-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/62657199-dd95-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/1e1eaeb0-dab5-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/a489013b-dab2-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-i2ib2
tmpfs 15448278 5 15448273 1% /var/lib/kubelet/pods/049d59d3-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 11 15448267 1% /var/lib/kubelet/pods/e42096b3-dab1-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/70379894-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-x0oc9
tmpfs 15448278 7 15448271 1% /var/lib/kubelet/pods/64428f9e-daad-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 5 15448273 1% /var/lib/kubelet/pods/051ee6f8-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/705ea9e4-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-de72y
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/717b54b7-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-xnayj
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/0507888f-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-rphkp
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/9553acc6-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-ktnvk
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/64428f9e-daad-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-jyg03
tmpfs 15448278 5 15448273 1% /var/lib/kubelet/pods/087c8706-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/0821289e-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/087c8706-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/adc06bf3-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/8c13eb1c-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/c48a04ca-dd95-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/967544e6-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-tnu0s
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/70d30696-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-tsbym
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/61c4aee2-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/fc267e59-dbf8-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-ktnvk
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/7051675a-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-7e03u
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/045510f5-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-7e03u
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/70f943f0-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-tsbym
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/6d8a5b0e-dccb-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-zyyq0
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/9578a9cd-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-j12r1
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/8d633140-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/df8da69b-dab1-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-ux6bn
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/099f72f9-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-cojhe
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/9796f3a2-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-tsbym
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/eb7d3c6a-dd95-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/66b9a75d-dd98-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/049d59d3-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-ktnvk
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/e700a7cf-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/70bc9993-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-c7gml
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/9ae5ddaa-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-gq8y5
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/e1ba154f-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/046545a1-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-7e03u
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/046df432-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-7e03u
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/588f7166-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 11 15448267 1% /var/lib/kubelet/pods/a489013b-dab2-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/fa00711a-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/c82524ee-daac-11e6-a503-42010a84014d/volumes/kubernetes.io~secret/default-token-bjls5
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/05e54acf-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/656b650b-dc0e-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/62d189b9-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/05bfb597-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-tsbym
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/044c3d95-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-shsmr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/b6c8c20e-dc10-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/09fa22f0-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-kjq40
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/95154ae3-daae-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-de72y
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/fa2d7dea-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/d777d5c6-dd98-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 15 15448263 1% /var/lib/kubelet/pods/4af7abd4-dbe4-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/REMOVED
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/dfaf7c72-dab1-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-xxsbb
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/083f941e-dab3-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-anvcr
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/8189d62d-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/c8274cef-daac-11e6-a503-42010a84014d/volumes/kubernetes.io~secret/default-token-bjls5
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/e0431378-dab1-11e6-9c9c-42010a84012a/volumes/kubernetes.io~secret/default-token-qdms9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/4af7abd4-dbe4-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-g7idw
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/7069d99f-dd94-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/c8334122-daac-11e6-a503-42010a84014d/volumes/kubernetes.io~secret/default-token-bjls5
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/f9754dd6-ddcd-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-rphkp
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/69ca64b2-ddce-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-rphkp
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/63da138f-ddce-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-rphkp
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/4cea0cc5-ddcb-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-j12r1
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/b9d8ac1e-dd99-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/27124bf1-dd9a-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-giqn9
/dev/sdc 655360 12 655348 1% /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/REMOVED
/dev/sdb 655360 12 655348 1% /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/REMOVED
/dev/sdd 655360 12 655348 1% /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/REMOVED
tmpfs 15448278 9 15448269 1% /var/lib/kubelet/pods/3723abd5-de17-11e6-808e-42010a84012a/volumes/kubernetes.io~secret/default-token-rphkp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment