Skip to content

Instantly share code, notes, and snippets.

@yuzutas0

yuzutas0/mysql_dump_tsv_to_bigquery_by_dataflow.py

Created June 22, 2019 05:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yuzutas0/d4f6a19eb9cbafb5415be2468f9e2e88 to your computer and use it in GitHub Desktop.
Save yuzutas0/d4f6a19eb9cbafb5415be2468f9e2e88 to your computer and use it in GitHub Desktop.
Download ZIP
ETL Logic: MySQL TSV Dump -> BigQuery by Python3.5 Dataflow
Raw
mysql_dump_tsv_to_bigquery_by_dataflow.py
import re
magic_word = '{{{{{converted_by_yuzutas0_dataflow}}}}}'
text = text.replace('\\\"', '\"\"').replace('\\\\', magic_word) \
.replace('\\\"', '\\').replace(magic_word, '\\\\')
text = re.sub(r'\\$', '', text, flags=re.MULTILINE)
if text.startswith('""') and not text.startswith('"""'):
text = text[1:]
if text.endswith('""') and not text.endswith('"""'):
text = text[0:-1]
return text
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment