Skip to content

Instantly share code, notes, and snippets.

@qqpann
Last active February 6, 2021 10:22
Show Gist options
  • Save qqpann/e3199f9420767c72e45be3c02922d1dd to your computer and use it in GitHub Desktop.
Save qqpann/e3199f9420767c72e45be3c02922d1dd to your computer and use it in GitHub Desktop.
BigQuery read and save with hashed name
from pathlib import Path
from hashlib import sha256
datadir = Path('data')
def get_df_from_bq(sql: str, **opt):
hashed = sha256(sql.encode("utf-8")).hexdigest()
cache = datadir / "cache"
cache.mkdir(exist_ok=True)
fname = cache / f"{hashed}.csv"
if fname.exists():
return pd.read_csv(fname, **opt)
else:
client = bigquery.Client()
df = client.query(sql).to_dataframe()
df.to_csv(fname, index=False)
return df
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment