Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to split a file into smaller chunks? — First published in fullweb.io issue #90

How to split a file into smaller chunks?

You may want to upload a 100GB file over an unstable network, or feed your scripts smaller inputs to load in RAM. In both cases, just splitting your file into smaller chunks is an easy solution.

Great news is Unix/Linux systems have the split utility already installed. And using it is simple as pie:

Cut a binary file into chunks of X bytes:

split -b X bigfile.avi

Cut a text file into chunks of N lines:

split -l N bigfile.csv

split being a Unix tool, you can actually pipe to it like this:

./gen_lotta_data.py | split -l 1000 -

It will save chunks into files named xaa, xab, xac, etc. Use the -a option to specify a filename suffix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment