Skip to content

Instantly share code, notes, and snippets.

@sadikovi

sadikovi/spark-parquet-writer-settings.scala

Last active March 13, 2023 07:35
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sadikovi/3fdf18040b4fad4c10ecd16e160cbe1b to your computer and use it in GitHub Desktop.
Save sadikovi/3fdf18040b4fad4c10ecd16e160cbe1b to your computer and use it in GitHub Desktop.
Download ZIP
Spark Parquet writer v1/v2 settings
Raw
spark-parquet-writer-settings.scala
sc.hadoopConfiguration.set("parquet.writer.version", "v1") // either "v1" or "v2"
// disable vectorized reading, does not support delta encoding
spark.conf.set("spark.sql.parquet.enableVectorizedReader", "false")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment