Skip to content

Instantly share code, notes, and snippets.

@PsyChip
Created July 20, 2018 11:54
Show Gist options
  • Save PsyChip/d797d05d16964a5830ed9a84696444bf to your computer and use it in GitHub Desktop.
Save PsyChip/d797d05d16964a5830ed9a84696444bf to your computer and use it in GitHub Desktop.
resumable gigantic csv importer for postgresql. csv column count must match with your table
#!/bin/bash
## usage:
## >import.sh filename.csv table database
folder=__csv
mkdir $folder
split -l 10000 $1 $folder/part
cd $folder
psql -c "copy $2 FROM 'partaa' DELIMITER ',' CSV HEADER;" $3;
rm partaa
for x in $(ls part*); do
echo "importing: $x";
psql -c "copy $2 FROM '$x' DELIMITER ',';" $3;
rm $x
echo "remaining blocks:";
ls | wc -l
done
echo 'completed';
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment