Skip to content

Instantly share code, notes, and snippets.

@holgerd77
Created November 27, 2014 12:07
Show Gist options
  • Save holgerd77/247068570ce145571359 to your computer and use it in GitHub Desktop.
Save holgerd77/247068570ce145571359 to your computer and use it in GitHub Desktop.
Farmsubsidy data (recipient, payment) CSV split script
#!/bin/bash
#Usage : ./split.sh COUNTRY_CODE FILE_TYPE_WITHOUT_ENDING LINES_PER_FILE
#Author : Holger Drewes
#Last edit : 2014-11-27
echo -e "Script for splitting payment or recipient files into chunks"
echo -e "***********************************************************\n"
#Check for number of arguments
if [ "$#" -ne 3 ]; then
echo "Usage: $0 COUNTRY_CODE FILE_TYPE_WITHOUT_ENDING LINES_PER_FILE"
exit 1
fi
country=$1
dir=./${country}/
filetype=$2
filepath=${dir}${filetype}.txt
lines=$3
tmpfile=tmp_file_split.txt
echo "Splitting file $filepath into chunked files with $lines lines..."
tail -n +2 ${filepath} | split -dl ${lines} - ${filetype}_FC --filter='cat > $FILE.txt'
echo "Adding CSV header to files..."
for file in ${filetype}_FC*
do
head -n 1 $filepath > $tmpfile
cat $file >> $tmpfile
mv -f $tmpfile $dir$file
echo "New file: $dir$file"
rm $file
done
echo "Processing finished."
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment