Skip to content

Instantly share code, notes, and snippets.

@gilangvperdana
Last active June 3, 2022 02:58
Show Gist options
  • Save gilangvperdana/8f1486f38aeaf1d3f7c5d7aa2c6994d2 to your computer and use it in GitHub Desktop.
Save gilangvperdana/8f1486f38aeaf1d3f7c5d7aa2c6994d2 to your computer and use it in GitHub Desktop.
Automation Delete Index ELK with Curator

Automation Delete Index ELK with Curator

If you have an issue that your Index getting bigger day by day, you can automation delete your OLD Index with Curator

Before

Before you want to try this, make sure you have a ELK Cluster before.

Installation

  • Install Dependencies
sudo apt install python3-pip
pip install elasticsearch-curator
  • Create a configuration
nano config.yml
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 10.0.0.71 #ip publish es
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Make an Delete Behaviour

nano delete_indices.yml
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 1 hours
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
      allow_ilm_indices: true
    filters:
    - filtertype: pattern
      kind: prefix
      value: ssh*
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y-%m-%d_%h'
      unit: hours
      unit_count: 1

Please crosscheck your configuration behaviour

  • Make sure value is same with Index Pattern on Index Management Kibana
  • Make sure timestring must be same with configuration on /etc/logstash/conf.d/input_output.conf on output section
  • Unit Hours & Unit Count can be interpreted remove index older than 1 hour

General Command

curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML

  • Execute our conf
curator --config config.yml delete_indices.yml

Finish!

Automate with Crontab

crontab -e
0 6 * * * /usr/local/bin/curator --config /home/user/config.yml /home/user/delete_indices.yml

Reference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment