Skip to content

Instantly share code, notes, and snippets.

@pat-humphreys
Last active August 18, 2016 15:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pat-humphreys/cc76fdac4c70ca98b1c9dd8f6e436d13 to your computer and use it in GitHub Desktop.
Save pat-humphreys/cc76fdac4c70ca98b1c9dd8f6e436d13 to your computer and use it in GitHub Desktop.

###Use Elastic Curator to cleanup ElasticSearch index backups in S3 ElasticSearch has a great index backup/restore API, which makes the whole process really simple. A single RESTful call creates an incremental backup of the indices and stores it to S3, simple!! Problem is they don’t give you an easy way to manage your backups, so you can end up with massive amounts of data in S3, we had over 8TB which we didn’t need. Enter Curator which allows you to manage indices and snapshots easily.

Install curator either on an ElasticSearch server or another server with access to it Remove "==3.5.1" if you want the latest version

pip install elasticsearch-curator==3.5.1

###Dry run to verify config (wont delete anything) curator --dry-run --config curator.yml --dry-run actionFile.yml

###Run and delete snapshots curator --config curator.yml --dry-run actionFile.yml

#Details of specif actions to carryout in order
actions:
1:
action: delete_snapshots
description: Delete index Snapshots over 3 days old
options:
repository: my-snapshots-repo
filters:
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 3
#Global curator config
client:
hosts: 10.1.1.1
port: 9200
timeout: 300
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment