Skip to content

Instantly share code, notes, and snippets.

@markwalkom
Last active April 29, 2022 10:23
Show Gist options
  • Save markwalkom/8a7201e3f6ea4354ae06 to your computer and use it in GitHub Desktop.
Save markwalkom/8a7201e3f6ea4354ae06 to your computer and use it in GitHub Desktop.
Reindexing Elasticsearch with Logstash 2.0
input {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
port => "9200"
index => "INDEXNAME_HERE"
size => 1000
scroll => "5m"
docinfo => true
scan => true
}
}
output {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
index => "%{[@metadata][_index]}"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}
stdout {
codec => "dots"
}
}
@JeremyColton
Copy link

JeremyColton commented Jul 25, 2016

Hi, I use the default index naming "logstash-" for a daily index. I have altered the number of shards from the default 5 to 1. I need to re-index my indices. I don't want to re-index into a new index eg "logstash-new-" but instead I want the existing indices to end up being spread across their single shard (instead of the current 5 shards per index).

How can I use this logstash script to do this?

Is there a better way to do this - eg re-index into new indices eg "logstash-new-", delete the original "logstash-" indices, then re-index back into "logstash-" from the new "logstash-new-" indices?

Many thanks.

@geekpete
Copy link

geekpete commented May 4, 2017

Reindex API is a nice option:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#_reindex_daily_indices

Also look into automatic scroll slicing that allows scrolls to be processed by multiple threads in parallel giving a nice speed boost.

@ksemaev
Copy link

ksemaev commented Feb 22, 2019

Can anybody please explain that scroll option? I do reindex with logstash and it loops endlessly - the data from source index is randomly duplicated to output

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment