Skip to content

Instantly share code, notes, and snippets.

@redmcg
Last active Nov 15, 2021
Embed
What would you like to do?
Bash script to show k8s PVC usage
#!/usr/bin/env bash
NODESAPI=/api/v1/nodes
function getNodes() {
kubectl get --raw $NODESAPI | jq -r '.items[].metadata.name'
}
function getPVCs() {
jq -s '[flatten | .[].pods[].volume[]? | select(has("pvcRef")) | '\
'{name: .pvcRef.name, capacityBytes, usedBytes, availableBytes, '\
'percentageUsed: (.usedBytes / .capacityBytes * 100)}] | sort_by(.name)'
}
function column() {
awk '{ for (i = 1; i <= NF; i++) { d[NR, i] = $i; w[i] = length($i) > w[i] ? length($i) : w[i] } } '\
'END { for (i = 1; i <= NR; i++) { printf("%-*s", w[1], d[i, 1]); for (j = 2; j <= NF; j++ ) { printf("%*s", w[j] + 1, d[i, j]) } print "" } }'
}
function defaultFormat() {
awk 'BEGIN { print "PVC 1K-blocks Used Available Use%" } '\
'{$2 = $2/1024; $3 = $3/1024; $4 = $4/1024; $5 = sprintf("%.0f%%",$5); print $0}'
}
function humanFormat() {
awk 'BEGIN { print "PVC Size Used Avail Use%" } '\
'{$5 = sprintf("%.0f%%",$5); printf("%s ", $1); system(sprintf("numfmt --to=iec %s %s %s | sed '\''N;N;s/\\n/ /g'\'' | tr -d \\\\n", $2, $3, $4)); print " " $5 }'
}
function format() {
jq -r '.[] | "\(.name) \(.capacityBytes) \(.usedBytes) \(.availableBytes) \(.percentageUsed)"' |
$format | column
}
if [ "$1" == "-h" ]; then
format=humanFormat
else
format=defaultFormat
fi
for node in $(getNodes); do
kubectl get --raw $NODESAPI/$node/proxy/stats/summary
done | getPVCs | format
@redmcg

This comment has been minimized.

Copy link
Owner Author

@redmcg redmcg commented May 8, 2019

Run with:
./kubedf

or with:
./kubedf -h

for human readable format.

@zenkovac

This comment has been minimized.

Copy link

@zenkovac zenkovac commented Jul 30, 2019

this is gold, thanks!

@voxmaster

This comment has been minimized.

Copy link

@voxmaster voxmaster commented Oct 22, 2019

Can you add -r as jq option in function getNodes() to aviod qoutes in output . Thanks!

@redmcg

This comment has been minimized.

Copy link
Owner Author

@redmcg redmcg commented Oct 22, 2019

@voxmaster Done. Thanks - I didn't know about that option.

@svanellewee

This comment has been minimized.

Copy link

@svanellewee svanellewee commented Nov 20, 2019

beautiful

@yourblizzx

This comment has been minimized.

Copy link

@yourblizzx yourblizzx commented Nov 22, 2019

Function getPVCs doesn't return anything, what could be the reason, is it coz of CSI plugin?

@lucax88x

This comment has been minimized.

Copy link

@lucax88x lucax88x commented Dec 18, 2019

Function getPVCs doesn't return anything, what could be the reason, is it coz of CSI plugin?

you need to have a kubectl proxy running.

@tomsherrod

This comment has been minimized.

Copy link

@tomsherrod tomsherrod commented Feb 26, 2020

kubectl proxy is running. Nothing returned from getPVCs.
jq -s '[flatten | .[].pods[].volume[]? | select(has("pvcRef"))]' returns nothing. I see the pod referenced in full json dump. No pvcRef.
Exec into the pod and see the mounted volume fine.
Tried it on kubernetes 1.14 and 1.15...nothing in stats/summary with pvcRef.
Using nfs-provisioner for dynamic provisioning.
Pointers to what I'm missing? I so want this to work!

@redmcg

This comment has been minimized.

Copy link
Owner Author

@redmcg redmcg commented Feb 27, 2020

You could try the following. Set up the following environment variables:

namespace=[namespace]
pod=[podname]

And then check your volume is listed when you run:
kubectl -n $namespace get pod $pod -o json | jq '.spec.volumes[]? | select(has("persistentVolumeClaim"))'

If not, then it's not recognised by Kubernetes as a PVC.

If it is, check the output of (edit: you'll need to have kubectl proxy running for this bit):

node=$(kubectl -n $namespace get pod $pod -o json | jq -r '.spec.nodeName')
volume=$(kubectl -n $namespace get pod $pod -o json | jq -r '.spec.volumes[]? | select(has("persistentVolumeClaim")) | .name')
KUBEAPI=127.0.0.1:8001/api/v1/nodes
curl -s $KUBEAPI/$node/proxy/stats/summary | jq -s '.[].pods[] | select(.podRef.name == "'$pod'") | .volume[]? | select(.name == "'$volume'")'

If it's listed, but doesn't have the 'pvcRef' like the example below:

{
  "time": "2020-02-27T08:06:36Z",
  "availableBytes": 8031363072,
  "capacityBytes": 8320901120,
  "usedBytes": 272760832,
  "inodesFree": 524057,
  "inodes": 524288,
  "inodesUsed": 231,
  "name": "data",
  "pvcRef": {
    "name": "pvcRefExample",
    "namespace": "default"
  }
}

then you might need to dig around the internals of Kubernetes, or possibly your provisioner to understand why.

@tomsherrod

This comment has been minimized.

Copy link

@tomsherrod tomsherrod commented Feb 27, 2020

@redmcg

This comment has been minimized.

Copy link
Owner Author

@redmcg redmcg commented Feb 27, 2020

If there's no output at all from the curl statement, you might want to try:
curl -i $KUBEAPI/$node/proxy/stats/summary

to include the response headers, or even:
curl -v $KUBEAPI/$node/proxy/stats/summary

for additional debug. This might help identify why you're not getting any output.

@tomsherrod

This comment has been minimized.

Copy link

@tomsherrod tomsherrod commented Feb 29, 2020

@guillaumedossantos

This comment has been minimized.

Copy link

@guillaumedossantos guillaumedossantos commented Apr 2, 2020

same as @tomsherrod except I use Rook Ceph with FlexVolumes.
I do not have any PvcRef with : curl -i $KUBEAPI/$node/proxy/stats/summary
Is it the driver that does not pass PvcRef along?
Can't test with CSI drivers for the moment, I'll let you know

@kramik1

This comment has been minimized.

Copy link

@kramik1 kramik1 commented Mar 22, 2021

So I adjusted your script to work for my Microk8s env and double checked the results. It is missing pvcref as well. But I think I know why...hostpath does not display them. Each volume manager has to implement metrics. Hostpath doesn't even show limits within the container themselves let alone so its all kind of moot for me I guess.

https://stackoverflow.com/questions/44718268/how-to-monitor-disk-usage-of-kubernetes-persistent-volumes

@mattwelke

This comment has been minimized.

Copy link

@mattwelke mattwelke commented Apr 21, 2021

Very useful, ty.

@brun0queiroz

This comment has been minimized.

Copy link

@brun0queiroz brun0queiroz commented Oct 15, 2021

For those who don't want to use kubectl proxy

#!/usr/bin/env bash

function getNodes() {
  kubectl get --raw=/api/v1/nodes | jq -r '.items[].metadata.name'
}

function getPVCs() {
  jq -s '[flatten | .[].pods[].volume[]? | select(has("pvcRef")) | '\
'{name: .pvcRef.name, capacityBytes, usedBytes, availableBytes, '\
'percentageUsed: (.usedBytes / .capacityBytes * 100)}] | sort_by(.name)'
}

function column() {
  awk '{ for (i = 1; i <= NF; i++) { d[NR, i] = $i; w[i] = length($i) > w[i] ? length($i) : w[i] } } '\
'END { for (i = 1; i <= NR; i++) { printf("%-*s", w[1], d[i, 1]); for (j = 2; j <= NF; j++ ) { printf("%*s", w[j] + 1, d[i, j]) } print "" } }'
}

function defaultFormat() {
  awk 'BEGIN { print "PVC 1K-blocks Used Available Use%" } '\
'{$2 = $2/1024; $3 = $3/1024; $4 = $4/1024; $5 = sprintf("%.0f%%",$5); print $0}'
}

function humanFormat() {
  awk 'BEGIN { print "PVC Size Used Avail Use%" } '\
'{$5 = sprintf("%.0f%%",$5); printf("%s ", $1); system(sprintf("numfmt --to=iec %s %s %s | sed '\''N;N;s/\\n/ /g'\'' | tr -d \\\\n", $2, $3, $4)); print " " $5 }'
}

function format() {
  jq '.[] | "\(.name) \(.capacityBytes) \(.usedBytes) \(.availableBytes) \(.percentageUsed)"' |
    sed 's/^"\|"$//g' |
    $format | column
}

if [ "$1" == "-h" ]; then
  format=humanFormat
else
  format=defaultFormat
fi

for node in $(getNodes); do
  kubectl get --raw=/api/v1/nodes/$node/proxy/stats/summary
done | getPVCs | format
@redmcg

This comment has been minimized.

Copy link
Owner Author

@redmcg redmcg commented Oct 18, 2021

@brun0queiroz That's much better. Thanks for sharing that. I've updated the gist

@leonK-BO

This comment has been minimized.

Copy link

@leonK-BO leonK-BO commented Oct 27, 2021

I am getting this error: parse error:

parse error: Invalid numeric literal at line 1, column 10
PVC Size Used Avail Use%

@redmcg

This comment has been minimized.

Copy link
Owner Author

@redmcg redmcg commented Oct 27, 2021

That's an error from jq. To find out why, you might want to view the individual responses. You can try something similar to what's described here:
https://gist.github.com/redmcg/60cfff7bca6f32969188008ad4a44c9a#gistcomment-3191345

@jessequinn

This comment has been minimized.

Copy link

@jessequinn jessequinn commented Nov 6, 2021

Linode LKE returns nothing. What could be the reason?

@redmcg

This comment has been minimized.

Copy link
Owner Author

@redmcg redmcg commented Nov 7, 2021

It will likely be a result of the csi driver in use:
https://kubernetes-csi.github.io/docs/drivers.html

The driver may not be providing k8s their metrics.

You could follow the instructions here to get a better picture:
https://gist.github.com/redmcg/60cfff7bca6f32969188008ad4a44c9a#gistcomment-3191345

@BLshlomo

This comment has been minimized.

Copy link

@BLshlomo BLshlomo commented Nov 12, 2021

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment