Skip to content

Instantly share code, notes, and snippets.

@iahmad-khan
Forked from mynameiswhm/reindex.sh
Created May 20, 2018 12:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save iahmad-khan/2b95fc6ee2ad814c563cf14af9559db7 to your computer and use it in GitHub Desktop.
Save iahmad-khan/2b95fc6ee2ad814c563cf14af9559db7 to your computer and use it in GitHub Desktop.
Bulk reindexing elasticsearch to another cluster using jq
#/bin/bash
if [ $# -lt 3 ]
then
echo "Usage: reindex.sh source_url target_url index [query]"
echo "Example: reindex.sh http://source.com:9200 http://target.com:9200 products_v1 '{\"query\":{\"range\":{\"mod_date\":{\"from\":\"2015-12-03T20:00:00\"}}}}'"
exit 1
fi
SOURCE_URL=$1
DESTINATION_URL=$2
index=$3
query=$4
document_type="default-type"
scroll_id=$(curl -s -XPOST $SOURCE_URL'/'$index'/_search?size=50&scroll=1m&search_type=scan' -d "$query" | jq '."_scroll_id"' | tr -d '"')
echo "scroll_id: $scroll_id"
page=1
while true
do
echo "page: $page"
page=$((page + 1))
file=/tmp/$document_type.json
curl -s $SOURCE_URL'/_search/scroll?scroll=1m' -d "$scroll_id" | jq -c '.hits.hits[] ._source' > $file
if [[ "$(wc -l $file | awk '{print $1}')" -eq "0" ]]
then
echo "finished"
exit 0
fi
cat $file | jq -c '{"index": {"_index": "'$index'", "_type": "'$document_type'", "_id": .id}}, .' | curl -s -XPOST $DESTINATION_URL/_bulk --data-binary @- > /dev/null
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment