Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Playing with kubectl output

Kubectl output options

Let's look at some basic kubectl output options.

Our intention is to list nodes (with their AWS InstanceId) and Pods (sorted by node).

We can start with:

kubectl get no

and

kubectl get po -o wide

Json and Jq

I've found the internal data structures easier to explore using the -o json output with jid and jq.

Once both jq and jid are installed (assuming OSX), we can quickly discover the data with the following command:

kubectl get no -o json | jid -q | pbcopy

This allows us to explore the json data interactively and keep our final jq query on the clipboard:

asciicast

note: jid currently implements it's own query parser to allow powerfull autocompletion, the drawback is a lack of support for all the jq constructs (i.e.: we have to specify an index for array elements during discovery).

As can be seen in the recording: once done with jid, getting rid of the index on the items array in jq, did gave us the full listing.

jq gives us a lot more power for example:

Boxing the result into it's own array and constructing a new object combining several nested attributes gives us the following query:

kubectl get no -o json | jq -r '[.items[] | {name:.metadata.name, id:.spec.externalID, unschedulable:.spec.unschedulable}]'

Here is how the above query was built up using jid and jq: asciicast

Converting the json array into a tabular output with jq can be done using @tsv as follows:

kubectl get no -o json | jq -r '.items[] | select(.spec.unschedulable!=true) | [.metadata.name,.spec.externalID] | @tsv'

Jq also allows us to sort:

kubectl get po -o json | jq -r '.items | sort_by(.spec.nodeName)[] | [.spec.nodeName,.metadata.name] | @tsv'

The input for the sort_by command must be an array, we iterate the elements after the sorting.

Custom Columns and Sorting

If all we need is a nicely formatted, sorted tabular report, kubectl has built-in support for powerfull sorting:

kubectl get po -o wide --sort-by=.spec.nodeName

Using jid to list pods sorted by node: asciicast

The usage of Custom Columns with the knowledge of the data structure gained from jid, is also much easier:

kubectl get no -o=custom-columns=NAME:.metadata.name,AWS-INSTANCE:.spec.externalID,UNSCHEDULABLE:.spec.unschedulable

Note: apart from using grep, there is no easy way to filter.

Golang Templates

If we do not wish to use jq (or have no access to jq) need filtering and powerfull output control, we may use Kubectl's built-in support for golang templates (inline or from a template file on disk):

kubectl get no -o go-template='{{range .items}}{{if .spec.unschedulable}}{{.metadata.name}} {{.spec.externalID}}{{"\n"}}{{end}}{{end}}'
or
kubectl get no -o go-template="{{range .items}}{{if .spec.unschedulable}}{{.metadata.name}} {{.spec.externalID}}:{{end}}{{end}}" | tr ":" "\n"

I could not find an easy way to print newline characters with inline golang template, so used a trick printing colons and using tr to convert colons to newlines.

JSONPath

Golang templates can be complicated and verbose - an alternative, if you are more familiar with jq-style queries, or awscli, is to use JSONPath.

kubectl get no -o jsonpath="{.items[?(@.spec.unschedulable)].metadata.name}"

Internally, this seems tightly coupled to the golang templates.

Kubectl supports a superset of JSONPath, with a special range keyword to iterate over ranges, using the same trick to add newlines:

kubectl get no -o jsonpath="{range.items[?(@.spec.unschedulable)]}{.metadata.name}:{end}" | tr ":" "\n"

More examples of using jsonpath can be found in the Kubernetes tests for the JSONPath utility

@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Feb 2, 2017

To list all services in a cluster and their nodeports:

kubectl get --all-namespaces svc -o json | jq -r '.items[] | [.metadata.name,([.spec.ports[].nodePort | tostring ] | join("|"))] | @csv'

Exercise: make it more concise using gotemplates

@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Mar 14, 2017

Quickly switch contexts and namespaces with these aliases

alias kswitch='kubectl config use-context'
alias knamespace='kubectl config set-context `kubectl config current-context` --namespace'

Get namespace of current context

export kcontext=`kubectl config current-context`
export kns=kubectl config view -o jsonpath="{.contexts[?(@.name==\"$kcontext\")].context.namespace}"

Can be added to bash prompt, if using powerline with following segment https://github.com/so0k/powerline-kubernetes

or blended into https://twitter.com/micahhausler/status/785833744863268871

NORMAL="\[\033[00m\]"
BLUE="\[\033[01;34m\]"
RED="\[\e[1;31m\]"
YELLOW="\[\e[1;33m\]"
GREEN="\[\e[1;32m\]"
PS1_WORKDIR="\w"
PS1_HOSTNAME="\h"
PS1_USER="\u"

__kube_ps1()
{
    CONTEXT=$(kubectl config current-context)
    NAMESPACE=$(kubectl config view -o jsonpath="{.contexts[?(@.name==\"${CONTEXT}\")].context.namespace}")
    if [ -z "$NAMESPACE"]; then
        NAMESPACE="default"
    fi
    if [ -n "$CONTEXT" ]; then
        case "$CONTEXT" in
          *prod*)
            echo "${RED}(⎈ ${CONTEXT} - ${NAMESPACE})"
            ;;
          *test*)
            echo "${YELLOW}(⎈ ${CONTEXT} - ${NAMESPACE})"
            ;;
          *)
            echo "${GREEN}(⎈ ${CONTEXT} - ${NAMESPACE})"
            ;;
        esac
    fi
}

export PROMPT_COMMAND='PS1="${GREEN}${PS1_USER}@${PS1_HOSTNAME}${NORMAL}:$(__kube_ps1)${BLUE}${PS1_WORKDIR}${NORMAL}\$ "'
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Mar 14, 2017

other handy aliases:

# get all services with in a cluster and the nodePorts they use (if any)
alias ksvc="kubectl get --all-namespaces svc -o json | jq -r '.items[] | [.metadata.name,([.spec.ports[].nodePort | tostring ] | join(\"|\"))] | @csv'"
# shortcuts for frequent kubernetes commands
alias kpods="kubectl get po"
alias kinspect="kubectl describe"
function krun() { name=$1; shift; image=$1; shift; kubectl run -it --generator=run-pod/v1 --image $image $name -- $@; }
function klogs() { kubectl logs $*;}
function kexec(){ pod=$1; shift; kubectl exec -it $pod -- $@; }
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Mar 17, 2017

sample of running a pipeline on every pod (in this case, es bound to primary interface, not 0.0.0.0):

export appname=elasticsearch
for p in `kubectl get po -l app=$appname -o name`;do echo $p; kubectl exec ${p:4} -- sh -c 'hostname -i | xargs -I {} wget -qO- {}:9200/| grep "\"name\""'; done

output

pod/1
  "name" : "Archer",
pod/2
  "name" : "Ogress",
pod/3
  "name" : "Silver Dagger",

(only works if all elasticsearch pods expose client services..)

@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Mar 21, 2017

Get status transitions of each pod (can use grep to filter down to terminated pods)

export jsonpath='{range .items[*]}{"\n"}{@.metadata.name}{range @.status.conditions[*]}{"\t"}{@.type}={@.status}{end}{end}' 
kubectl get po -ao jsonpath="$jsonpath" && echo

source

see also:

Delete failed jobs (thnks @RobinSE)

$ kubectl delete pod $(kubectl get pods --show-all -o go-template='{{range .items}}{{if eq .status.phase "Failed"}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Mar 29, 2017

Run a command on every cluster node,

for example, get uptime of all nodes (depends on your ssh setup, see below)

while read ip; do echo $ip; ssh $ip uptime </dev/null; done < <(kubectl get no -o jsonpath='{range.items[*].status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}')

note: ssh eats stdin by default, have to redirect it's stdin (</dev/null) to ensure the while loop can continue reading from stdin

note: ssh setup - if you are using a bastion host for your k8s nodes in subnet 10.100.*, you can use following ssh

Host <bastion-host-ip>
  User core
  IdentityFile ~/.ssh/<bastion-host-key>.pem

Host 10.100.*
  User core
  ProxyCommand ssh -A -W %h:%p <bastion-host-ip>
  IdentityFile ~/.ssh/<bastion-host-key>.pem

kubectl delete pod stays in terminating status

Is docker ps hanging on any node? Where is my container (maybe you force deleted the pod in k8s, but it seems it is still alive somewhere...:

while read ip; do echo $ip; ssh $ip "docker ps | grep my-container" < /dev/null; done < <(kubectl get no -o json | jq -r '.items[].status.addresses[] | select(.type=="InternalIP") | .address')

this time with jq as an alternative

What else is running on this mis-behaving node?

kubectl get po -o wide | grep ip-10-100-10-25
@tuannvm

This comment has been minimized.

Copy link

tuannvm commented Jul 13, 2017

Get k8s node's internal IPs:

kubectl get no -o json | jq -r '.items[].status.addresses[] | select(.type=="InternalIP") | .address
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Aug 4, 2017

Get Secret:

secret_name=test
DOCKER_REGISTRY=registry.example.com
kubectl get secret ${secret_name}  -o jsonpath="{['data']['\.dockercfg']}" | base64 -D | jq .

# or just read it into 2 env vars
read DOCKER_USERNAME DOCKER_PASSWORD <<<$(kubectl get secret ${secret_name} -o jsonpath="{['data']['\.dockercfg']}" | base64 -D | jq  -r ".[\"${DOCKER_REGISTRY}\"] | [.username,.password] | @tsv")
@TerraTech

This comment has been minimized.

Copy link

TerraTech commented Aug 28, 2017

Original:

Kubectl supports a superset of JSONPath, with a special range keyword to iterate over ranges, using the same trick to add newlines:

kubectl get no -o jsonpath="{range.items[?(@.spec.unschedulable)]}{.metadata.name}:{end}" | tr ":" "\n"

@so0k This should work for adding newlines:

kubectl get no -o jsonpath='{range.items[?(@.spec.unschedulable)]}{.metadata.name}{"\n"}{end}'

I'm currently using this for finding PVs that have Released PVCs:

kubectl get pv -o jsonpath='{range.items[?(@.status.phase=="Released")]}{.metadata.name}{"\n"}{end}'
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Aug 28, 2017

@ghulevishal

This comment has been minimized.

Copy link

ghulevishal commented Oct 30, 2017

Get PV

kubectl get pv -o json | jq -r '.items | sort_by(.spec.capacity.storage)[]|[.metadata.name,.spec.capacity.storage]| @tsv'
@ghulevishal

This comment has been minimized.

Copy link

ghulevishal commented Oct 30, 2017

Get Pod on specific node.

kubectl get po -o json | jq -r '.items | sort_by(.spec.nodeName)[]|select(.spec.nodeName=="node3")|[.metadata.name,.spec.nodeName]| @tsv'
@ghulevishal

This comment has been minimized.

Copy link

ghulevishal commented Oct 30, 2017

Get sorted list of nodes with respect to Memory Capacity.

kubectl get no -o json | jq -r '.items | sort_by(.status.capacity.memory)[]|[.metadata.name,.status.capacity.memory]| @tsv'
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Jan 10, 2018

based on kubernetes/kubernetes#55051 (comment) (and tested on k8s 1.7)

kubectl get po -a --all-namespaces -o json | jq  '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | "kubectl delete po \(.metadata.name) --n \(.metadata.namespace)"' | xargs -n 1 bash -c

trying out fieldSelector see valid fields - to delete all non-running pods...

kubectl get po -a --all-namespaces -o json --field-selector="status.phase!=Running" | jq  '.items[]  | "kubectl delete po \(.metadata.name) --n \(.metadata.namespace)"' | xargs -n 1 bash -c
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Apr 23, 2018

Get podcount per node, in ascending order

kubectl get po -o json --all-namespaces | jq '.items | group_by(.spec.nodeName) | map({"nodeName": .[0].spec.nodeName, "count": length}) | sort_by(.count)'

do note that for very large clusters it might be faster to use jq inputs - https://github.com/stedolan/jq/wiki/Cookbook#processing-a-large-number-of-lines-or-json-entities

@geoand

This comment has been minimized.

Copy link

geoand commented May 8, 2018

Get the display name of all the clusterserviceclass objects that are provided by the Ansible Service Broker (Openshift specific):

kubectl get clusterserviceclass -o json | jq -r '.items[].spec | select(.clusterServiceBrokerName=="ansible-service-broker") | .externalMetadata.displayName'
@kevinbringard

This comment has been minimized.

Copy link

kevinbringard commented May 22, 2018

FYI, It looks like someone created a fork of jid and added jq filtering queries: https://github.com/fiatjaf/jiq

@pracucci

This comment has been minimized.

Copy link

pracucci commented Jun 19, 2018

For each pod, list whether it containers run on a read-only root filesystem or not:

kubectl get pods --all-namespaces -o go-template --template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range .spec.containers}}    read-only: {{if .securityContext.readOnlyRootFilesystem}}{{printf "\033[32m%t\033[0m" .securityContext.readOnlyRootFilesystem}} {{else}}{{printf "\033[91m%s\033[0m" "false"}}{{end}} ({{.name}}){{"\n"}}{{end}}{{"\n"}}{{end}}'
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Jun 29, 2018

for any resource type, labels can be printed as columns using the -L flag:

kubectl get no -Lbeta.kubernetes.io/instance-type -Lfailure-domain.beta.kubernetes.io/zone
NAME                                           STATUS    ROLES     AGE       VERSION   INSTANCE-TYPE   ZONE
ip-10-51-....ap-southeast-1.compute.internal   Ready     node      39d       v1.9.3    m4.xlarge       ap-southeast-1c
ip-10-51-....ap-southeast-1.compute.internal   Ready     node      43d       v1.9.3    m4.xlarge       ap-southeast-1c
ip-10-51-....ap-southeast-1.compute.internal   Ready     node      43d       v1.9.3    m4.xlarge       ap-southeast-1a
ip-10-51-....ap-southeast-1.compute.internal   Ready     master    43d       v1.9.3    m4.large        ap-southeast-1a
ip-10-51-....ap-southeast-1.compute.internal   Ready     master    43d       v1.9.3    m4.large        ap-southeast-1b
ip-10-51-....ap-southeast-1.compute.internal   Ready     node      43d       v1.9.3    m4.xlarge       ap-southeast-1b
ip-10-51-....ap-southeast-1.compute.internal   Ready     master    43d       v1.9.3    m4.large        ap-southeast-1c
$ kubectl get po -l release=panama-dev -Lcomponent
NAME                                             READY     STATUS    RESTARTS   AGE       COMPONENT
panama-dev-amqp-68fbd99f9c-4w2ph                 1/1       Running   0          42d       amqp
panama-dev-apidoc-6bc7c87c9d-s4t7x               2/2       Running   0          31d       apidoc
panama-dev-artemis-56c786b6c8-bgn5f              1/1       Running   0          31d       artemis
panama-dev-artemis-56c786b6c8-p6hfv              1/1       Running   0          31d       artemis
panama-dev-asteria-56cfdbf769-rjclt              2/2       Running   0          31d       asteria
@jhliptak

This comment has been minimized.

Copy link

jhliptak commented Sep 5, 2018

go-templates can have newlines in them using the println function.

For example, to get all Users in all namespaces (assuming you have permissions):
kubectl get rolebindings --all-namepsaces -o go-template --template='{{range .items}}{{println}}{{.metadata.namespace}}={{range .subjects}}{{if eq .kind \"User\"}}{{.name}} {{end}}{{end}}{{end}}'
will print the namespaces followed by a space separated list of users.

@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Nov 16, 2018

$ jq --version
jq-1.6

$ k get secret -o json | jq '.items[] | [{"secret":.metadata.name,"data":.data | values | to_entries | map({"key":.key,"value":.value | @base64d}) | from_entries}]'
@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Jan 13, 2019

With base64d function exposed to go template

template='{{ range $k, $v := .data }}{{ $v | base64decode}}{{"\n"}}{{end}}'

Source - https://twitter.com/oldmanuk/status/1065358163497676801?s=21

@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Jan 21, 2019

This may be more useful for exploration than jid- https://github.com/antonmedv/fx

@StevenACoffman

This comment has been minimized.

Copy link

StevenACoffman commented Mar 2, 2019

You may find that https://github.com/ashleyschuett/kubernetes-secret-decode allows nicer output:

k get secret -o yaml | ksd 

It will automatically decode the base64 output

@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Apr 3, 2019

Read Datadog annotations (this is mostly jq features)

$ k get po -l app=envoy -o json | jq '.items[0].metadata.annotations | with_entries(select(.key | startswith("ad.datadoghq.com/"))) | map_values(fromjson)'
{
  "ad.datadoghq.com/envoy.check_names": [
    "envoy"
  ],
  "ad.datadoghq.com/envoy.init_configs": [
    {}
  ],
  "ad.datadoghq.com/envoy.instances": [
    {
      "stats_url": "http://%%host%%:8002/stats"
    }
  ],
  "ad.datadoghq.com/envoy.logs": [
    {
      "source": "envoy",
      "service": "ingress"
    }
  ]
}

or kops channel addons configuration

$ kubectl get ns kube-system -o json | jq '.metadata.annotations | with_entries(select(.value | contains("addons"))) | map_values(fromjson | .version)'
{
  "addons.k8s.io/authentication.aws": "0.3.0",
  "addons.k8s.io/bootstrap.swatmobile.io": "0.0.9",
  "addons.k8s.io/core.addons.k8s.io": "1.4.0",
  "addons.k8s.io/coredns.addons.k8s.io": "1.2.6-kops.1",
  "addons.k8s.io/dns-controller.addons.k8s.io": "1.11.0",
  "addons.k8s.io/fluent-bit-logging.swatmobile.io": "0.0.2",
  "addons.k8s.io/heptio-auth-aws.swatmobile.io": "0.0.3",
  "addons.k8s.io/ingress.swatmobile.io": "0.0.10",
  "addons.k8s.io/limit-range.addons.k8s.io": "1.5.0",
  "addons.k8s.io/networking.cilium.io": "v1.0-kops.2",
  "addons.k8s.io/rbac.addons.k8s.io": "1.8.0",
  "addons.k8s.io/state-metrics.swatmobile.io": "0.0.2",
  "addons.k8s.io/storage-aws.addons.k8s.io": "1.7.0"
}
@kivagant-ba

This comment has been minimized.

Copy link

kivagant-ba commented Apr 19, 2019

Find all resources requests and limits per pod

kubectl get pods -o json -n kube-system | jq -r '.items[] | .metadata.name + " \n Req. RAM: " + .spec.containers[].resources.requests.memory + " \n Lim. RAM: " + .spec.containers[].resources.limits.memory + " \n Req. CPU: " + .spec.containers[].resources.requests.cpu + " \n Lim. CPU: " + .spec.containers[].resources.limits.cpu + " \n Req. Eph. DISK: " + .spec.containers[].resources.requests["ephemeral-storage"] + " \n Lim. Eph. DISK: " + .spec.containers[].resources.limits["ephemeral-storage"] + "\n"'

coredns-7bcbfc4774-7sxg2
 Req. RAM: 70Mi
 Lim. RAM: 170Mi
 Req. CPU: 100m
 Lim. CPU:
 Req. Eph. DISK: 100Mi
 Lim. Eph. DISK: 768Mi
...

Get all requests or limits from a namespace

echo "\nRAM Requests TOTAL:" && kubectl describe namespace kube-system | grep 'requests.memory' && echo "\nRAM Requests:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.requests.memory + " | " + .metadata.name'

echo "\nRAM Limits TOTAL:" && kubectl describe namespace kube-system | grep 'limits.memory' &&  echo "\nRAM Limits:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.limits.memory + " | " + .metadata.name'

echo "\nCPU Requests TOTAL:" && kubectl describe namespace kube-system | grep 'requests.cpu' &&  echo "\nCPU Requests:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.requests.cpu + " | " + .metadata.name'

echo "\nCPU Limits TOTAL:" && kubectl describe namespace kube-system | grep 'limits.cpu' &&  echo "\nCPU Limits:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.limits.cpu + " | " + .metadata.name'

echo "\nEph. DISK Requests TOTAL:" && kubectl describe namespace kube-system | grep 'requests.ephemeral-storage' && echo "\nEph. DISK Requests:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.requests["ephemeral-storage"] + " | " + .metadata.name'

echo "\nEph. DISK Limits TOTAL:" && kubectl describe namespace kube-system | grep 'limits.ephemeral-storage' && echo "\nEph. DISK Limits:\n" && kubectl get pods -o json -n kube-system | jq -r '.items[] | .spec.containers[].resources.limits["ephemeral-storage"] + " | " + .metadata.name'


RAM Requests TOTAL:
 requests.memory               3640Mi   16Gi

RAM Requests:

70Mi | coredns-7bcbfc4774-7sxg2
...

Do some operations for pods using simple grep

NS=kube-system; kubectl get pod -n $NS | grep "ContainerCreating \|Init:0/1\|ContainerCreating\|ImagePullBackOff" | awk '{print $1}'|xargs -I % sh -c "kubectl -n $NS describe pod % "
@steinrh

This comment has been minimized.

Copy link

steinrh commented Sep 18, 2019

Hi!
I would like to ask if there is a query with jsonpath that I could get the running time from a pod in seconds?
Using this query I will get the complete date:
$ kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'
alertmanager-main-0 2019-XX-XXT16:34:01Z
What I expect is to write a query and get:
alertmanager-main-0 36567(as seconds running)

Is that possible to do?

@Guillaume-Mayer

This comment has been minimized.

Copy link

Guillaume-Mayer commented Oct 29, 2019

What about piping json logs to jq?
I tried kubectl logs -f mypod | jq but it shows nothing until I ctrl+c it

@so0k

This comment has been minimized.

Copy link
Owner Author

so0k commented Oct 30, 2019

@steinrh - I think all the age fields are more polished in kubectl (same with trying to get total allocated vs total available resources of nodes, only the kubectl describe has a nice result there)

@Guillaume-Mayer ... that's a tricky one, maybe jq needs to use streaming flag to read stdin the way kubectl logs expects it?

@Guillaume-Mayer

This comment has been minimized.

Copy link

Guillaume-Mayer commented Oct 30, 2019

@so0k - It should be --unbuffered or --stream but none of those worked for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.