Skip to content

Instantly share code, notes, and snippets.

@dipankarmazumdar

dipankarmazumdar/ingest_iceberg.py

Created April 25, 2024 17:36
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save dipankarmazumdar/aa0c616284fa69a1cc0713820b560404 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save dipankarmazumdar/aa0c616284fa69a1cc0713820b560404 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
ingest_iceberg.py
spark.sql(
"""CREATE OR REPLACE TEMPORARY VIEW salesview USING csv
OPTIONS (path "file:/home/docker/All_Data_Aldi.csv", header true)"""
)
spark.sql("INSERT INTO hdfs_catalog.retail_ice SELECT * FROM salesview")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment