Skip to content

Instantly share code, notes, and snippets.

View dev-mansonthomas's full-sized avatar

Thomas Manson dev-mansonthomas

View GitHub Profile
@shantanuo
shantanuo / mysql_to_big_query.sh
Last active September 14, 2022 07:12
Copy MySQL table to big query. If you need to copy all tables, use the loop given at the end. Exit with error code 3 if blob or text columns are found. The csv files are first copied to google cloud before being imported to big query.
#!/bin/sh
TABLE_SCHEMA=$1
TABLE_NAME=$2
mytime=`date '+%y%m%d%H%M'`
hostname=`hostname | tr 'A-Z' 'a-z'`
file_prefix="trimax$TABLE_NAME$mytime$TABLE_SCHEMA"
bucket_name=$file_prefix
splitat="4000000000"
bulkfiles=200