Skip to content

Instantly share code, notes, and snippets.

@djouallah
Created October 25, 2023 12:50
Show Gist options
  • Save djouallah/daefc9e57a1a735663e241dc92db4d7a to your computer and use it in GitHub Desktop.
Save djouallah/daefc9e57a1a735663e241dc92db4d7a to your computer and use it in GitHub Desktop.
write to Onelake using Python
%%time
!pip install -q duckdb
!pip install -q deltalake
import duckdb
from deltalake.writer import write_deltalake
from trident_token_library_wrapper import PyTridentTokenLibrary
aadToken = PyTridentTokenLibrary.get_access_token("storage")
sf =1
for x in range(0, sf) :
con=duckdb.connect()
con.sql('PRAGMA disable_progress_bar;SET preserve_insertion_order=false')
con.sql(f"CALL dbgen(sf={sf} , children ={sf}, step = {x})")
for tbl in ['nation','region','customer','supplier','lineitem','orders','partsupp','part'] :
df=con.sql(f"SELECT * FROM {tbl}").arrow()
write_deltalake(f"abfss://xxxxxx@onelake.dfs.fabric.microsoft.com/yyyyyy.Lakehouse/Tables/{tbl}",\
df,mode='overwrite',overwrite_schema=True,storage_options={"bearer_token": aadToken, "use_fabric_endpoint": "true"})
con.close()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment