Skip to content

Instantly share code, notes, and snippets.

@agalea91

agalea91/s3_gs_bigquery_upload.sh

Last active October 28, 2020 05:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save agalea91/f8b11d3f0785b5f33e0359f56f088532 to your computer and use it in GitHub Desktop.
Save agalea91/f8b11d3f0785b5f33e0359f56f088532 to your computer and use it in GitHub Desktop.
Download ZIP
Upload from s3 to gs and then to bigquery
Raw
s3_gs_bigquery_upload.sh
aws s3 cp s3://my_bucket/my_table /tmp/my_table --recursive \
&& gcloud config set account <name>@<project>.iam.gserviceaccount.com \
&& gcloud config set project <project> \
&& gcloud auth activate-service-account <name>@<project>.iam.gserviceaccount.com --key-file /my_path_to/google_credentials.json
&& gsutil cp -r /tmp/my_table gs://bucket_name/my_table \
&& bq mk --dataset my_dataset \
&& bq mk --table --schema my_table_schema.json my_dataset.my_table \
&& bq query --nouse_legacy_sql 'delete from my_dataset.my_table where <date_range_where_clause>;' \
&& bq load --source_format=CSV --quote "" -F='\t' my_dataset.my_table gs://bucket_name/my_table/0000_part_00 \
&& bq load --source_format=CSV --quote "" -F='\t' my_dataset.my_table gs://bucket_name/my_table/0001_part_00 \
&& bq load --source_format=CSV --quote "" -F='\t' my_dataset.my_table gs://bucket_name/my_table/0002_part_00 \
&& bq load --source_format=CSV --quote "" -F='\t' my_dataset.my_table gs://bucket_name/my_table/0003_part_00 \
&& gsutil rm -r gs://bucket_name/my_table \
&& rm -rf /tmp/my_table
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment