Skip to content

Instantly share code, notes, and snippets.

@ML-engineer
Created July 4, 2022 22:08
Show Gist options
  • Star 22 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save ML-engineer/e28ae87e830ca7f8e594bce6ca33e38c to your computer and use it in GitHub Desktop.
Save ML-engineer/e28ae87e830ca7f8e594bce6ca33e38c to your computer and use it in GitHub Desktop.
Read BQ table to DuckDB directly from storage read api
import duckdb
from google.cloud import bigquery
bqclient = bigquery.Client()
table = bigquery.TableReference.from_string(
"bigquery-public-data.utility_us.country_code_iso"
)
rows = bqclient.list_rows(table)
country_code_iso = rows.to_arrow(create_bqstorage_client=True)
cursor = duckdb.connect()
print(cursor.execute('SELECT * FROM country_code_iso').fetchall())
@datadutch
Copy link

do you have a requirements.txt to accompany this script ? the google.cloud/bigquery requirements keep causing incompatibility issues

@ML-engineer
Copy link
Author

ML-engineer commented Jul 5, 2022

@bidutch I'm using virtualenv with
pip install duckdb
pip install pyarrow
pip install google-cloud-bigquery
pip install google-cloud-bigquery-storage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment