Using Curator to reindex your Elasticsearch indices

Using Curator to reindex your Elasticsearch indices.md

i. Installation

First, you need to add the Public Signing Key

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Add your distro's correct path to /etc/apt/sources.list.d/ with a .list suffix

deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main

Update the repository with sudo apt update

ii. Config files Create your Curator config, config.yml in /etc/curator/config.yml

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 127.0.0.1 //This should be the address of your Elasticsearch instance
  port: 9200 //The port of your Elasticseach instance, 9200 is the default value
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO //CRITICAL, ERROR, WARNING, INFO, DEBUG, the default is INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

After this, you'll need an action file. Action files have the following structure

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: ACTION1
    description: OPTIONAL DESCRIPTION
    options:
      option1: value1
      ...
      optionN: valueN
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: *first*
      filter_element1: value1
      ...
      filter_elementN: valueN
    - filtertype: *second*
      filter_element1: value1
      ...
      filter_elementN: valueN
  2:
    action: ACTION2
    description: OPTIONAL DESCRIPTION
    options:
      option1: value1
      ...
      optionN: valueN
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: *first*
      filter_element1: value1
      ...
      filter_elementN: valueN
    - filtertype: *second*
      filter_element1: value1
      ...
      filter_elementN: valueN
  3:
    action: ACTION3
    ...
  4:
    action: ACTION4
    ...

So if you want for example reindex your daily indices to monthy indeces your action file will look something like this reindex.yml file

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: "Reindex indexname-2019.10.{30,31} into new-indexname-2019.10"
    action: reindex
    options:
      disable_action: False
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: ['indexname-2019.10.*']
        dest:
          index: new-indexname-2019.10
    filters:
    - filtertype: none

If your new index is not exist yet, you must create it first with

$ curl -XPUT http://YOUR-ES-INSTANCE/indexname-2019.10 -d '{"settings": {"number_of_shards": 5, "number_of_replicas": 1}}'

To run your action file, you can use

curator --config curator.yml reindex.yml

Note that this could take quite long, depending your indices.