Skip to content

Instantly share code, notes, and snippets.

@mynameiswhm
Created December 4, 2015 05:18
Show Gist options
  • Save mynameiswhm/9437d663bc80accfeeaf to your computer and use it in GitHub Desktop.
Save mynameiswhm/9437d663bc80accfeeaf to your computer and use it in GitHub Desktop.
Bulk reindexing elasticsearch to another cluster using jq
#/bin/bash
if [ $# -lt 3 ]
then
echo "Usage: reindex.sh source_url target_url index [query]"
echo "Example: reindex.sh http://source.com:9200 http://target.com:9200 products_v1 '{\"query\":{\"range\":{\"mod_date\":{\"from\":\"2015-12-03T20:00:00\"}}}}'"
exit 1
fi
SOURCE_URL=$1
DESTINATION_URL=$2
index=$3
query=$4
document_type="default-type"
scroll_id=$(curl -s -XPOST $SOURCE_URL'/'$index'/_search?size=50&scroll=1m&search_type=scan' -d "$query" | jq '."_scroll_id"' | tr -d '"')
echo "scroll_id: $scroll_id"
page=1
while true
do
echo "page: $page"
page=$((page + 1))
file=/tmp/$document_type.json
curl -s $SOURCE_URL'/_search/scroll?scroll=1m' -d "$scroll_id" | jq -c '.hits.hits[] ._source' > $file
if [[ "$(wc -l $file | awk '{print $1}')" -eq "0" ]]
then
echo "finished"
exit 0
fi
cat $file | jq -c '{"index": {"_index": "'$index'", "_type": "'$document_type'", "_id": .id}}, .' | curl -s -XPOST $DESTINATION_URL/_bulk --data-binary @- > /dev/null
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment