Skip to content

Instantly share code, notes, and snippets.

@djouallah
Created June 28, 2023 02:23
Show Gist options
  • Save djouallah/01f73d2f88bbcdf67a3ffc8056ae38c4 to your computer and use it in GitHub Desktop.
Save djouallah/01f73d2f88bbcdf67a3ffc8056ae38c4 to your computer and use it in GitHub Desktop.
#V order thing you can ignore those two lines
spark.conf.set("spark.sql.parquet.vorder.enabled", "true")
spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true")
#Load from the default lakehouse, make sure you click on the pin
from pyspark.sql.types import *
df = spark.read.option("header", "true").format("csv").load("Files/csv/*.csv")
df.write.mode("overwrite").format("delta").save("Tables/tablecsv")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment