Skip to content

Instantly share code, notes, and snippets.

@jhinds
Created May 18, 2017 18:38
Show Gist options
  • Save jhinds/7c69c79237460fb5c09d10116125cec6 to your computer and use it in GitHub Desktop.
Save jhinds/7c69c79237460fb5c09d10116125cec6 to your computer and use it in GitHub Desktop.
Splitting a CSV & keeping headers
#!/bin/bash
# file that needs to be split
filename="big_file.csv"
# directory for split files
split_dir="splits"
# string to prepend files
split_prepend="small_file-"
# get the header from the first file
head -1 $filename > header.csv
# split the file into files with 100000 rows
split -l 100000 $filename ${split_dir}/${split_prepend}
# loop through split files & combine with header then clean up
for small_file in ${split_dir}/*
do
cat header.csv $small_file > $small_file.csv
rm $small_file
done
rm header.csv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment