Skip to content

Instantly share code, notes, and snippets.

@jatorre
Created April 12, 2021 08:17
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jatorre/2127b8486d474706807935a9083c9a6a to your computer and use it in GitHub Desktop.
Save jatorre/2127b8486d474706807935a9083c9a6a to your computer and use it in GitHub Desktop.
Script to upload GEO files to bigQuery using GDAL
#!/bin/bash
if [ -f "$1" ]; then
for fullpath in "$1"
do
filename="${fullpath##*/}" # Strip longest match of */ from start
dir="${fullpath:0:${#fullpath} - ${#filename}}" # Substring from 0 thru pos of filename
base="${filename%.[^.]*}" # Strip shortest match of . plus at least one non-dot char from end
ext="${filename:${#base} + 1}" # Substring from len of base thru end
if [[ -z "$base" && -n "$ext" ]]; then # If we have an extension and no base, it's really the base
base=".$ext"
ext=""
fi
done
/Library/Frameworks/GDAL.framework/Programs/ogr2ogr -f CSV -dialect sqlite -sql "select AsGeoJSON(geometry,6) AS geom, * from $base" temp$base.csv $1
bq load --autodetect --source_format=CSV $2_temp temp$base.csv
bq query --use_legacy_sql=false "CREATE TABLE $2 AS SELECT * EXCEPT (geom), SAFE.ST_GEOGFROMGEOJSON(geom) as geom FROM $2_temp"
bq rm -f $2_temp
rm -rf temp$base.csv
else
echo "Please specify a valid file to import"
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment