#Git notes
Warning : Support for isplay of git notes has been dropped by github : https://github.com/blog/707-git-notes-display
Resource : https://vimeo.com/34273537
##Add
git notes add
iljau
rvaidya
/ database_to_parquet.py
Last active
November 5, 2023 11:15
Dump database table to parquet file using sqlalchemy and fastparquet. Useful for loading large tables into pandas / Dask, since read_sql_table will hammer the server with queries if the # of partitions/chunks is high. Using this you write a temp parquet file, then use read_parquet to get the data into a DataFrame
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pandas as pd | |
| import numpy as np | |
| import fastparquet | |
| from sqlalchemy import create_engine, schema, Table | |
| # Copied from pandas with modifications | |
| def __get_dtype(column, sqltype): | |
| import sqlalchemy.dialects as sqld |
