Skip to content

Instantly share code, notes, and snippets.

@herrbuerger
Last active October 28, 2015 15:51
Show Gist options
  • Save herrbuerger/10961577 to your computer and use it in GitHub Desktop.
Save herrbuerger/10961577 to your computer and use it in GitHub Desktop.
Rotating snapshots for Elasticsearch

Rotationg snapshots for Elasticsearch (poor-man's version)

This is a very basic method to do rotating snapshots for Elasticsearch. For this to work you will need to have jq installed (http://stedolan.github.io/jq/).

PLEASE, PLEASE, PLEASE don't put this blindly in your commandline and execute it :)

curl -s -S -XGET "localhost:9200/_snapshot/my_s3_repository/_all?pretty=true" | jq '.snapshots[] | .snapshot + " " + .end_time' | sed 's/^.\(.*\).$/\1/' | sort -k 2 -r | awk '{ if (NR > 1) { system("curl -XDELETE " "localhost:9200/_snapshot/my_s3_repository/"$1) } } END { system("curl -XPUT " "localhost:9200/_snapshot/my_s3_repository/`date +\%s`") }'

Details

curl -s -S -XGET "localhost:9200/_snapshot/my_s3_repository/_all?pretty=true"

Fetch a list of all snapshots currently available in your repository.

jq '.snapshots[] | .snapshot + " " + .end_time'

Since the above command returns a JSON object, we fetch the relevant information with jq, i.e. the name of the snapshots and the time when they were finished.

sed 's/^.\(.*\).$/\1/'

jq returns a string with quotes at the beginning and the end, since they interfere with the next awk command, we will get rid of them (there is probably an easier method for this).

sort -k 2 -r

Sort the output in a descending order based on the second column, i.e. date

 awk '{ if (NR > 1) { system("curl -XDELETE " "localhost:9200/_snapshot/my_s3_repository/"$1) } } END { system("curl -XPUT " "localhost:9200/_snapshot/my_s3_repository/`date +\%s`") }'

Ok, this is the interesting part. awk now loops through the backups and deletes every one after the first position (NR > 1). If you want to keep more backups, you will need to change this number.

Once awk is finished with that, it creates a new snapshot with a timestamp.

I'm running this script with @daily in CRON.

@z0mbix
Copy link

z0mbix commented Aug 1, 2014

Use the raw option for jq (-r) to avoid the extra sed command

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment