Velero memory usage analysis

velero-memory-analysis.md

Setup port-forwarding for the velero metric server address

$ kubectl --namespace velero portforward deploy/velero 8085:8085

Scrapping Velero memory metrics

$ while true; do curl http://localhost:8085/metrics | grep -E \
'^velero_backup_tarball_size_bytes|^go_memstats|^go_threads|^process_resident_memory_bytes|^process_virtual_memory_bytes' \
>> ~/devnull/velero-mem-analysis/velero-go-memstats.txt; \
echo "---" >> ~/devnull/velero-mem-analysis/velero-go-memstats.txt; \
sleep 5; \
done

Data processing

Get metric names

cat velero-go-memstats.txt | awk '{print $1}' | sort -u > keys.txt

Get individual metrics

#! /bin/bash

while read p; do
	grep -w $p $2 | awk '{print $2}' > $p-values.txt;
done < $1

Save the above script as ./process.sh and run it as

$ ./process.sh keys.txt velero-go-memstats.txt

Aggregation

Aggregate each metric into a common .csv file to generate charts