Skip to content

Instantly share code, notes, and snippets.

@markwalkom

markwalkom/logstash.conf

Last active April 29, 2022 10:23
Show Gist options
  • Star 45 You must be signed in to star a gist
  • Fork 9 You must be signed in to fork a gist
  • Save markwalkom/8a7201e3f6ea4354ae06 to your computer and use it in GitHub Desktop.
Save markwalkom/8a7201e3f6ea4354ae06 to your computer and use it in GitHub Desktop.
Download ZIP
Reindexing Elasticsearch with Logstash 2.0
Raw
logstash.conf
input {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
port => "9200"
index => "INDEXNAME_HERE"
size => 1000
scroll => "5m"
docinfo => true
scan => true
}
}
output {
elasticsearch {
hosts => [ "HOSTNAME_HERE" ]
index => "%{[@metadata][_index]}"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}
stdout {
codec => "dots"
}
}
@geekpete
Copy link

geekpete commented May 4, 2017

Reindex API is a nice option:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#_reindex_daily_indices

Also look into automatic scroll slicing that allows scrolls to be processed by multiple threads in parallel giving a nice speed boost.

@ksemaev
Copy link

ksemaev commented Feb 22, 2019

Can anybody please explain that scroll option? I do reindex with logstash and it loops endlessly - the data from source index is randomly duplicated to output

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment