How to split a file into smaller chunks?
You may want to upload a 100GB file over an unstable network, or feed your scripts smaller inputs to load in RAM. In both cases, just splitting your file into smaller chunks is an easy solution.
Great news is Unix/Linux systems have the
split utility already installed. And using it is simple as pie:
Cut a binary file into chunks of X bytes:
split -b X bigfile.avi
Cut a text file into chunks of N lines:
split -l N bigfile.csv
split being a Unix tool, you can actually pipe to it like this:
./gen_lotta_data.py | split -l 1000 -
It will save chunks into files named
xac, etc. Use the
-a option to specify a filename suffix.