Skip to content

Instantly share code, notes, and snippets.

@so0k
Last active September 11, 2024 08:36
Show Gist options
  • Save so0k/42313dbb3b547a0f51a547bb968696ba to your computer and use it in GitHub Desktop.
Save so0k/42313dbb3b547a0f51a547bb968696ba to your computer and use it in GitHub Desktop.
Playing with kubectl output

Kubectl output options

Let's look at some basic kubectl output options.

Our intention is to list nodes (with their AWS InstanceId) and Pods (sorted by node).

We can start with:

kubectl get no

and

kubectl get po -o wide

Json and Jq

I've found the internal data structures easier to explore using the -o json output with jid and jq.

Once both jq and jid are installed (assuming OSX), we can quickly discover the data with the following command:

kubectl get no -o json | jid -q | pbcopy

This allows us to explore the json data interactively and keep our final jq query on the clipboard:

asciicast

note: jid currently implements it's own query parser to allow powerfull autocompletion, the drawback is a lack of support for all the jq constructs (i.e.: we have to specify an index for array elements during discovery).

As can be seen in the recording: once done with jid, getting rid of the index on the items array in jq, did gave us the full listing.

jq gives us a lot more power for example:

Boxing the result into it's own array and constructing a new object combining several nested attributes gives us the following query:

kubectl get no -o json | jq -r '[.items[] | {name:.metadata.name, id:.spec.externalID, unschedulable:.spec.unschedulable}]'

Here is how the above query was built up using jid and jq: asciicast

Converting the json array into a tabular output with jq can be done using @tsv as follows:

kubectl get no -o json | jq -r '.items[] | select(.spec.unschedulable!=true) | [.metadata.name,.spec.externalID] | @tsv'

Jq also allows us to sort:

kubectl get po -o json | jq -r '.items | sort_by(.spec.nodeName)[] | [.spec.nodeName,.metadata.name] | @tsv'

The input for the sort_by command must be an array, we iterate the elements after the sorting.

Custom Columns and Sorting

If all we need is a nicely formatted, sorted tabular report, kubectl has built-in support for powerfull sorting:

kubectl get po -o wide --sort-by=.spec.nodeName

Using jid to list pods sorted by node: asciicast

The usage of Custom Columns with the knowledge of the data structure gained from jid, is also much easier:

kubectl get no -o=custom-columns=NAME:.metadata.name,AWS-INSTANCE:.spec.externalID,UNSCHEDULABLE:.spec.unschedulable

Note: apart from using grep, there is no easy way to filter.

Golang Templates

If we do not wish to use jq (or have no access to jq) need filtering and powerfull output control, we may use Kubectl's built-in support for golang templates (inline or from a template file on disk):

kubectl get no -o go-template='{{range .items}}{{if .spec.unschedulable}}{{.metadata.name}} {{.spec.externalID}}{{"\n"}}{{end}}{{end}}'
or
kubectl get no -o go-template="{{range .items}}{{if .spec.unschedulable}}{{.metadata.name}} {{.spec.externalID}}:{{end}}{{end}}" | tr ":" "\n"

I could not find an easy way to print newline characters with inline golang template, so used a trick printing colons and using tr to convert colons to newlines.

JSONPath

Golang templates can be complicated and verbose - an alternative, if you are more familiar with jq-style queries, or awscli, is to use JSONPath.

kubectl get no -o jsonpath="{.items[?(@.spec.unschedulable)].metadata.name}"

Internally, this seems tightly coupled to the golang templates.

Kubectl supports a superset of JSONPath, with a special range keyword to iterate over ranges, using the same trick to add newlines:

kubectl get no -o jsonpath="{range.items[?(@.spec.unschedulable)]}{.metadata.name}:{end}" | tr ":" "\n"

More examples of using jsonpath can be found in the Kubernetes tests for the JSONPath utility

@so0k
Copy link
Author

so0k commented Jan 13, 2019

With base64d function exposed to go template

template='{{ range $k, $v := .data }}{{ $v | base64decode}}{{"\n"}}{{end}}'

Source - https://twitter.com/oldmanuk/status/1065358163497676801?s=21

@so0k
Copy link
Author

so0k commented Jan 21, 2019

This may be more useful for exploration than jid- https://github.com/antonmedv/fx

@StevenACoffman
Copy link

You may find that https://github.com/ashleyschuett/kubernetes-secret-decode allows nicer output:

k get secret -o yaml | ksd 

It will automatically decode the base64 output

@so0k
Copy link
Author

so0k commented Apr 3, 2019

Read Datadog annotations (this is mostly jq features)

$ k get po -l app=envoy -o json | jq '.items[0].metadata.annotations | with_entries(select(.key | startswith("ad.datadoghq.com/"))) | map_values(fromjson)'
{
  "ad.datadoghq.com/envoy.check_names": [
    "envoy"
  ],
  "ad.datadoghq.com/envoy.init_configs": [
    {}
  ],
  "ad.datadoghq.com/envoy.instances": [
    {
      "stats_url": "http://%%host%%:8002/stats"
    }
  ],
  "ad.datadoghq.com/envoy.logs": [
    {
      "source": "envoy",
      "service": "ingress"
    }
  ]
}

or kops channel addons configuration

$ kubectl get ns kube-system -o json | jq '.metadata.annotations | with_entries(select(.value | contains("addons"))) | map_values(fromjson | .version)'
{
  "addons.k8s.io/authentication.aws": "0.3.0",
  "addons.k8s.io/bootstrap.swatmobile.io": "0.0.9",
  "addons.k8s.io/core.addons.k8s.io": "1.4.0",
  "addons.k8s.io/coredns.addons.k8s.io": "1.2.6-kops.1",
  "addons.k8s.io/dns-controller.addons.k8s.io": "1.11.0",
  "addons.k8s.io/fluent-bit-logging.swatmobile.io": "0.0.2",
  "addons.k8s.io/heptio-auth-aws.swatmobile.io": "0.0.3",
  "addons.k8s.io/ingress.swatmobile.io": "0.0.10",
  "addons.k8s.io/limit-range.addons.k8s.io": "1.5.0",
  "addons.k8s.io/networking.cilium.io": "v1.0-kops.2",
  "addons.k8s.io/rbac.addons.k8s.io": "1.8.0",
  "addons.k8s.io/state-metrics.swatmobile.io": "0.0.2",
  "addons.k8s.io/storage-aws.addons.k8s.io": "1.7.0"
}

@kivagant-ba
Copy link

Find all resources requests and limits per pod

kubectl get pods -o json -n kube-system | jq -r '.items[] | .metadata.name + " \n Req. RAM: " + .spec.containers[].resources.requests.memory + " \n Lim. RAM: " + .spec.containers[].resources.limits.memory + " \n Req. CPU: " + .spec.containers[].resources.requests.cpu + " \n Lim. CPU: " + .spec.containers[].resources.limits.cpu + " \n Req. Eph. DISK: " + .spec.containers[].resources.requests["ephemeral-storage"] + " \n Lim. Eph. DISK: " + .spec.containers[].resources.limits["ephemeral-storage"] + "\n"'

coredns-7bcbfc4774-7sxg2
 Req. RAM: 70Mi
 Lim. RAM: 170Mi
 Req. CPU: 100m
 Lim. CPU:
 Req. Eph. DISK: 100Mi
 Lim. Eph. DISK: 768Mi
...

Get all requests or limits from a namespace

echo "\nRAM Requests TOTAL:" && kubectl describe namespace kube-system | grep 'requests.memory' && echo "\nRAM Requests:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.requests.memory + " | " + .metadata.name'

echo "\nRAM Limits TOTAL:" && kubectl describe namespace kube-system | grep 'limits.memory' &&  echo "\nRAM Limits:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.limits.memory + " | " + .metadata.name'

echo "\nCPU Requests TOTAL:" && kubectl describe namespace kube-system | grep 'requests.cpu' &&  echo "\nCPU Requests:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.requests.cpu + " | " + .metadata.name'

echo "\nCPU Limits TOTAL:" && kubectl describe namespace kube-system | grep 'limits.cpu' &&  echo "\nCPU Limits:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.limits.cpu + " | " + .metadata.name'

echo "\nEph. DISK Requests TOTAL:" && kubectl describe namespace kube-system | grep 'requests.ephemeral-storage' && echo "\nEph. DISK Requests:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.requests["ephemeral-storage"] + " | " + .metadata.name'

echo "\nEph. DISK Limits TOTAL:" && kubectl describe namespace kube-system | grep 'limits.ephemeral-storage' && echo "\nEph. DISK Limits:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.limits["ephemeral-storage"] + " | " + .metadata.name'


RAM Requests TOTAL:
 requests.memory               3640Mi   16Gi

RAM Requests:

70Mi | coredns-7bcbfc4774-7sxg2
...

Do some operations for pods using simple grep

NS=kube-system; kubectl get pod -n $NS | grep "ContainerCreating \|Init:0/1\|ContainerCreating\|ImagePullBackOff" | awk '{print $1}'|xargs -I % sh -c "kubectl -n $NS describe pod % "

@Guillaume-Mayer
Copy link

What about piping json logs to jq?
I tried kubectl logs -f mypod | jq but it shows nothing until I ctrl+c it

@so0k
Copy link
Author

so0k commented Oct 30, 2019

@steinrh - I think all the age fields are more polished in kubectl (same with trying to get total allocated vs total available resources of nodes, only the kubectl describe has a nice result there)

@Guillaume-Mayer ... that's a tricky one, maybe jq needs to use streaming flag to read stdin the way kubectl logs expects it?

@Guillaume-Mayer
Copy link

@so0k - It should be --unbuffered or --stream but none of those worked for me

@kksudo
Copy link

kksudo commented Dec 13, 2019

@kivagant-ba

Find all resources requests and limits per pod

kubectl get pods -o json -n kube-system | jq -r '.items[] | .metadata.name + " \n Req. RAM: " + .spec.containers[].resources.requests.memory + " \n Lim. RAM: " + .spec.containers[].resources.limits.memory + " \n Req. CPU: " + .spec.containers[].resources.requests.cpu + " \n Lim. CPU: " + .spec.containers[].resources.limits.cpu + " \n Req. Eph. DISK: " + .spec.containers[].resources.requests["ephemeral-storage"] + " \n Lim. Eph. DISK: " + .spec.containers[].resources.limits["ephemeral-storage"] + "\n"'

This recursively will duplicate containers and pods, you can check it, just add container name.

kubectl get pods -o json -n kube-system | jq -r '.items[] | .metadata.name + " \n Container name: " + .spec.containers[].name + " \n Req. RAM: " + .spec.containers[].resources.requests.memory + " \n Lim. RAM: " + .spec.containers[].resources.limits.memory + " \n Req. CPU: " + .spec.containers[].resources.requests.cpu + " \n Lim. CPU: " + .spec.containers[].resources.limits.cpu + " \n Req. Eph. DISK: " + .spec.containers[].resources.requests["ephemeral-storage"] + " \n Lim. Eph. DISK: " + .spec.containers[].resources.limits["ephemeral-storage"] + "\n"'

Unfortunately I did not find how to remove duplicates (

@so0k
Copy link
Author

so0k commented Jan 7, 2020

if you still have 1.11 clusters, k get apiservices looks very different from 1.13+ clusters..

before:

$ k get apiservice
NAME                                   CREATED AT
v1.                                    2019-02-04T04:02:51Z
v1.apps                                2019-02-04T04:02:51Z
v1.authentication.k8s.io               2019-02-04T04:02:52Z
v1.authorization.k8s.io                2019-02-04T04:02:52Z
v1.autoscaling                         2019-02-04T04:02:52Z
v1.batch                               2019-02-04T04:02:52Z
v1.monitoring.coreos.com               2019-07-03T07:28:49Z
...

with jq -r and column -t:

$ k get apiservice -o json | jq -r '.items[] | [.metadata.name,(if .spec.service.name == null then "Local" else .spec.service.name end),.status.conditions[].type,.metadata.creationTimestamp] | @tsv' | column -t
v1.                                   Local                              Available  2019-02-04T04:02:51Z
v1.apps                               Local                              Available  2019-02-04T04:02:51Z
v1.authentication.k8s.io              Local                              Available  2019-02-04T04:02:52Z
v1.authorization.k8s.io               Local                              Available  2019-02-04T04:02:52Z
v1.autoscaling                        Local                              Available  2019-02-04T04:02:52Z
v1.batch                              Local                              Available  2019-02-04T04:02:52Z
v1.monitoring.coreos.com              Local                              Available  2019-07-03T07:28:49Z
v1.networking.k8s.io                  Local                              Available  2019-02-04T04:02:52Z
v1.rbac.authorization.k8s.io          Local                              Available  2019-02-04T04:02:52Z
v1.storage.k8s.io                     Local                              Available  2019-02-04T04:02:52Z
v1beta1.extensions                    Local                              Available  2019-02-04T04:02:52Z
v1beta1.external.metrics.k8s.io       datadog-cluster-agent-metrics-api  Available  2019-12-12T00:36:22Z
v1beta1.metrics.k8s.io                metrics-server                     Available  2019-12-20T01:17:22Z
v1beta1.policy                        Local                              Available  2019-02-04T04:02:52Z
...

@so0k
Copy link
Author

so0k commented Jan 13, 2020

Inspect a TLS secret

$ k get secret
NAME                          TYPE                                  DATA   AGE
my-certs                      kubernetes.io/tls                     3      273d

k get secret my-certs -o json | jq -r .data[\"tls.crt\"] | base64 -D | openssl x509 -in /dev/stdin -text -noout

@so0k
Copy link
Author

so0k commented Mar 5, 2020

converting a configmap to a secret, you want to base64 encode all values - using yq to convert yaml to json for jq and back to yaml:

yq r secrets.yaml data -j | jq 'map_values(@base64)' | yq r -

this one liner does not change the kind to secret nor does it add the type: Opaque fields required for secrets

EDIT: or reading a secret from the cluster

$ k get secret my-secrets -o json | jq -r '.data | map_values(@base64d)'

@boomkap
Copy link

boomkap commented Mar 25, 2020

How do i output the value of a field that has a . in the name when using a go-template. For e.g. the Quota object has fields with names like requests.cpu, requests.memory.

I am trying to pull quota allocations vs used for memory, cpu & storage using a go-template.

The closest I got is using the following:

oc get quota -o go-template --template='{{range .items}}{{.status}}{{range .hard}}{{end}}{{end}}' -n <namespace>

This produced the following output. As you can see, this is not ideal for further analysis.

map[hard:map[requests.cpu:5 requests.memory:150Gi] used:map[requests.cpu:1650m requests.memory:841Mi]]

I am assuming there is a way to escape the . in the name so the template engine does not interpret it as a subtree/leaf node.

Any ideas?

@so0k
Copy link
Author

so0k commented Apr 30, 2020

@boomkap - sorry, I didn't use quota - did you try\ as escape character?

to check image used by containers in pod:

k get po -o custom-columns=name:.metadata.name,status:.status.phase,image:'.spec.containers[0].image'
name                 status    image
frontend             Running   1234567.dkr.ecr.ap-southeast-1.amazonaws.com/frontend:0.1.0-058d8d4
...

@boomkap
Copy link

boomkap commented Apr 30, 2020

@so0k - does not work with \ as an escape character. Throws the following error

template: output1: bad character U+005c

@venkatalolla
Copy link

@boomkap I think this what you are looking for, a command to output the total number of Pods running, CPU request/limits and Memory requests/limits for each projects in a cluster.

oc get quota --all-namespaces -o=custom-columns=Project:.metadata.namespace,TotalPods:.status.used.pods,TotalCPURequest:.status.used.requests'\.'cpu,TotalCPULimits:.status.used.limits'\.'cpu,TotalMemoryRequest:.status.used.requests'\.'memory,TotalMemoryLimit:.status.used.limits'\.'memory

🙂

@vic501
Copy link

vic501 commented Aug 16, 2020

kubectl get no -o json | jq -r '.items | sort_by(.status.capacity.memory)[]|[.metadata.name,.status.capacity.memory]| @TSV'

how to display in MB or GB, currently is displaying in kb. ?

Thanks.

@tuxknight
Copy link

save ConfigMap into files, per file each key,using key as filename.

cm_name=config
ns=default
files=$(kubectl get cm $cm_name -n $ns -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}')
for f in $(echo $files);do
  kubectl get cm $cm_name -n $ns -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v}}{{end}}{{end}}' > $f 2>/dev/null
done

@berttejeda
Copy link

berttejeda commented Jan 8, 2021

Use jq to display Limits/Allocation Ratio for a given Kubernetes namespace quota:

kubectl get quota -n my-namespace -o json | jq -r "$(curl -ks https://gist.githubusercontent.com/berttejeda/e89da293d9b3ed8183948caac2bdd09c/raw/9dc2617baaa7f389ad8cbd92d3a7049169ad0cbc/k8s.quotas.jq)"

Sample output:

[
  {
    "namespace": "my-namespace",
    "quotaname": "default-12a3b",
    "limitsCpu": "360",
    "limitsCpu_used": "9",
    "limitsCpu_Allocation_Ratio": 2.5,
    "limitsMemory": "414Gi",
    "limitsMemory_used": "20Gi",
    "limitsMemory_Allocation_Ratio": 4.830917874396135,
    "limitsPod": "225",
    "limitsPods_used": "13",
    "limitsPod_Allocation_Ratio": 5.777777777777778
  }
]

@Leo7654
Copy link

Leo7654 commented Feb 25, 2021

Sum up all requested CPUs

k get pods -o json | jq -r '[.items[] | .spec.containers[].resources.requests.cpu | rtrimstr("m") | tonumber] | add'

@waddles
Copy link

waddles commented Apr 29, 2021

I also found this deep dive on using Go template files to render the output of kubectl commands (oc builds on kubectl). There is nothing OpenShift about this, it would have been better named as "Kubectl and Go Templates workshop"

https://github.com/brandisher/openshift-and-gotemplates-workshop

@charandas
Copy link

I had to add a -A and select on top of what @Leo7654 recommended, to get across all namespaces, and filter out empty results.

k get pods -A -o json | jq -r '[.items[] | .spec.containers[].resources.requests.cpu | rtrimstr("m") | select(length > 0) | tonumber] | add'

@ak2766
Copy link

ak2766 commented Nov 27, 2021

I've got a large number of pods and would like to see the output sorted a certain way. I'd like to sort the output based on multiple fields when there's a tie.

Is this possible using --sort-by?

I'm currently using bash scripts to do this but was hoping that a newer version (my cluster is still at v1.18.10) of Kubernetes can do this natively.

@morhook
Copy link

morhook commented Dec 17, 2021

@Guillaume-Mayer you should do the following command to fix that:

stdbuf -o0 kubectl logs -f mypod | jq

@SrikantPatil88
Copy link

Hi Folks,

I am trying to fetch external LB names only but condition is not working as expected,
Can someone help me to get the data of external LB only.

kubectl get services --all-namespaces -o json | jq -r '.items[] | select(.spec.type=="LoadBalancer")| select (.spec | with_entries(select(.key | contains("loadBalancerSourceRanges")))) | [.metadata.name] | @TSV'

Getting all LB not running second condition

kubectl get services --all-namespaces -o json |jq '.items[*].metadata.annotations | with_entries(select(.key | startswith("networking.gke.io/"))) | map_values(fromjson)'

Thanks in advance

@vincentgna
Copy link

vincentgna commented Nov 18, 2022

nodes sorted by lowest pod count using https://stackoverflow.com/questions/58201823/kubernetes-display-current-pods-vs-capacity-with-kubectl

kubectl get po -A -o json | jq -r '.items | sort_by(.spec.nodeName)[] | [.spec.nodeName,.metadata.name] | @csv' | awk -F, '{arr[$1]++}END{for (a in arr) print a, arr[a]}' | sort -nk2

Or using jq only

kubectl get po -A -o json | jq -r '[.items | sort_by(.spec.nodeName)[] | {"node":.spec.nodeName,"pod":.metadata.name}] | [group_by(.node)[] | {"node":.[0].node,"count":length}] | sort_by(.count)[] | [.node,.count] | @tsv'

@SilvM
Copy link

SilvM commented Dec 16, 2022

worth mentioning you can also just hit enter while editing the command:

❯ k get nodes -lnode-role.kubernetes.io/master= -o go-template='{{ range .items }}
quote> {{ index .metadata.labels "kubernetes.io/hostname"}}{{end}}'

ip-10-0-1-1
ip-10-0-1-2
ip-10-0-1-3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment