Skip to content

Instantly share code, notes, and snippets.

@peterwwillis
Last active March 27, 2024 14:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save peterwwillis/df23a7ed8dbfd1dde2ac79293b3a268c to your computer and use it in GitHub Desktop.
Save peterwwillis/df23a7ed8dbfd1dde2ac79293b3a268c to your computer and use it in GitHub Desktop.
Kubernetes debugging tips

Log Files

Nodes

Control Plane log files

  • /var/log/kube-apiserver.log - API Server, responsible for serving the API
  • /var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions
  • /var/log/kube-controller-manager.log - a component that runs most Kubernetes built-in controllers, with the notable exception of scheduling (the kube-scheduler handles scheduling).

Worker Node log files

  • /var/log/kubelet.log - logs from the kubelet, responsible for running containers on the node
  • /var/log/kube-proxy.log - logs from kube-proxy, which is responsible for directing traffic to Service endpoints

Tools


Kubectl

Display extra debugging information

Add the -v option to your kubectl commands. Provides the equivalent curl command used to query the API server.

$ kubectl get secrets -v6

Recreate API calls

Use the get --raw mode to query the API using Kubectl.

$ kubectl get --raw '/api/v1/namespaces/default/secrets?labelSelector=name%3Dcert-manager%2Cowner%3Dhelm&limit=500'

Check the version

Confirm the version of your kubectl client and k8s server API version.

$ kubectl version
Client Version: v1.28.8
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.8-gke.1067004

Get kubelet logs from a specific node

$ kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"

Query parameter options:

Option Description
boot boot show messages from a specific system boot
pattern pattern filters log entries by the provided PERL-compatible regular expression
query query specifies services(s) or files from which to return logs (required)
sinceTime an RFC3339 timestamp from which to show logs (inclusive)
untilTime an RFC3339 timestamp until which to show logs (inclusive)
tailLines specify how many lines from the end of the log to retrieve; the default is to fetch the whole log

Example of a more complex query:

$ kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error"

Helm

Display extra debugging information

Add the -v option to your helm commands. Provides the equivalent curl command used to query the API server.

$ helm list -v6

Guides

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment