Skip to content

Instantly share code, notes, and snippets.

@cuonghuynh

cuonghuynh/guide.md

Last active October 1, 2018 15:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cuonghuynh/17819347e922162db4a7d88942aa8c65 to your computer and use it in GitHub Desktop.
Save cuonghuynh/17819347e922162db4a7d88942aa8c65 to your computer and use it in GitHub Desktop.
Download ZIP
How to Import a large CSV file to our storage?
Raw
guide.md

Chunk data

If you have a large CSV file, you very likely can't store it all in an array in memory before you start inserting the data into the DB, so you need to be inserting the data in chunks as you read it from the CSV file.

Queue chunks

You would be lost the data if the process unfortinately broken.

Log errors

You should know why it is broken. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment