Skip to content

Instantly share code, notes, and snippets.

@jeffwong
Last active December 16, 2015 12:59
Show Gist options
  • Save jeffwong/5438468 to your computer and use it in GitHub Desktop.
Save jeffwong/5438468 to your computer and use it in GitHub Desktop.
Break a file into chunks
#!/bin/bash
if [ $# -lt 3 ] ;
then
echo "Usage: $0 inputfile output_prefix lines_per_chunk [header=1]"
exit
fi
header=1
if [ $# -eq 4 ] ;
then
header=$4
fi
lines=$(wc -l $1|awk '{print($1)}')
numchunks=$(($lines/$3))+1
START=1
END=numchunks
for ((i=$START; i<=$END; i++))
do
if [ $header -eq 0 ];
then
startchunk=$((($i-1)*$3+1))
else
startchunk=$((($i-1)*$3+2))
fi
endchunk="$(($startchunk + $3 - 1))p"
if [ $header -eq 0 ];
then
sed -n $startchunk,$endchunk $1 > "$2_chunk$i"
else
sed -n -e $startchunk,$endchunk -e 1p $1 > "$2_chunk$i"
fi
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment