Skip to content

Instantly share code, notes, and snippets.

@ML-engineer
Created July 4, 2022 22:08
Show Gist options
  • Save ML-engineer/e28ae87e830ca7f8e594bce6ca33e38c to your computer and use it in GitHub Desktop.
Save ML-engineer/e28ae87e830ca7f8e594bce6ca33e38c to your computer and use it in GitHub Desktop.
Read BQ table to DuckDB directly from storage read api
import duckdb
from google.cloud import bigquery
bqclient = bigquery.Client()
table = bigquery.TableReference.from_string(
"bigquery-public-data.utility_us.country_code_iso"
)
rows = bqclient.list_rows(table)
country_code_iso = rows.to_arrow(create_bqstorage_client=True)
cursor = duckdb.connect()
print(cursor.execute('SELECT * FROM country_code_iso').fetchall())
@ML-engineer
Copy link
Author

ML-engineer commented Jul 5, 2022

@bidutch I'm using virtualenv with
pip install duckdb
pip install pyarrow
pip install google-cloud-bigquery
pip install google-cloud-bigquery-storage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment