Prometheus DB Backup using TSDB Snapshot

prometheus_db_backup_using_Snapshot.md

Get Token for API Authentication

oc whoami -t

Get Prometheus Route URL

oc get route -n openshift-monitoring | grep -i prometheus

Run sample curl request

curl -ks -H 'Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4' 'https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/query?query=ALERTS' | python -m json.tool

Create TSDB Snapshot

curl -X 'POST' -ks -H 'Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4' 'https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot' | python -m json.tool

You might get an error, so you first need to enable Admin APIs

{
    "error": "Admin APIs are disabled",
    "message": "Admin APIs are disabled",
    "code": 14
}

Enable AdminAPI

oc -n openshift-monitoring patch prometheus k8s \
  --type merge --patch '{"spec":{"enableAdminAPI":true}}'

Verify Admin API is enabled

oc describe  po prometheus-k8s-1 | grep -i admin
      --web.enable-admin-api

Hit TSDB snapshot API to take snapshot

curl -X 'POST' -ks -H 'Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4' 'https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v2/admin/tsdb/snapshot'
 | python -m json.tool

{
    "name": "20210512T162601Z-33415dbd315ae6af"
}

Find the snapshot and copy it to locally.

• The default folder is /prometheus/snapshots/ but you can find the data folder by finding the --storage.tsdb.path config in your deployment.

curl -X 'POST' -ks -H 'Authorization: Bearer 0za4LjX9xPcqDjhWaufkgcQGo4grqA7ws4zvHrqgfY4' 'https://prometheus-k8s-openshift-monitoring.apps.ocp4.cp4d.com/api/v1/admin/tsdb/snapshot' | python -m json.tool

oc -n openshift-monitoring exec -it prometheus-k8s-0 -c prometheus -- /bin/sh -c "ls /prometheus/snapshots/20210512T162601Z-33415dbd315ae6af"

Copy the Snapshot locally

oc project openshift-monitoring
oc rsync prometheus-k8s-0:/prometheus/snapshots/ /home/prometheus