Skip to content

Instantly share code, notes, and snippets.

@romiras

romiras/es_bulk_upload.sh

Last active December 8, 2021 16:43
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save romiras/dc96cdebf3ae5d3f1ab5f6f55e5c10fd to your computer and use it in GitHub Desktop.
Save romiras/dc96cdebf3ae5d3f1ab5f6f55e5c10fd to your computer and use it in GitHub Desktop.
Download ZIP
Script for uploading 30 K dataset into ES
Raw
es_bulk_upload.sh
split --verbose -l1000 dataset-bulk-30k.ndjson bulk.
for f in bulk.??; do echo $f; curl -i -X POST localhost:9200/_bulk -H "Content-Type: application/x-ndjson" --data-binary @$f; done
@romiras
Copy link
Author

romiras commented Nov 11, 2021

Split 30K ndjson dataset into chunks 1000 per file, then import into ES with cURL.

@romiras
Copy link
Author

romiras commented Dec 8, 2021

For uploading to ElasticSearch that requires authentication you can add a parameter -u "$ES_MASTER_USER:$ES_MASTER_PW" to curl command above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment