Skip to content

Instantly share code, notes, and snippets.

View vadivelselvaraj's full-sized avatar

Vadivel vadivelselvaraj

View GitHub Profile
@vadivelselvaraj
vadivelselvaraj / compact.py
Last active November 10, 2022 04:20
CompactParquetFiles
# Read the S3 folder as glue dynamic data frames
input_dyf = glueContext.create_dynamic_frame_from_options("s3", {
"paths": [ inputPath ],
"recurse": True,
"groupFiles": "inPartition"
},
format = "parquet"
)
# Repartition them as required