Forked from looselycoupled/gist:4f23a6ed698b9c62387c596f5613fd6a
Created
November 18, 2021 07:47
-
-
Save chrisdlees/67cc29acda08bb924aa606e0e7a1901b to your computer and use it in GitHub Desktop.
Bash CSV 'split' Example
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # The following will split a CSV (file.csv) into multiple parts of 1 million lines each | |
| # with each part having its own header. | |
| # | |
| # PREFIX denotes the filename to use for the parts. A number will be added to the end. | |
| tail -n +2 file.csv | | |
| split -d -l 1000000 - --filter='sh -c "{ head -n1 file.csv; cat; } > $FILE"' PREFIX |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment