Skip to content

Instantly share code, notes, and snippets.

@mikaelhg

mikaelhg/zstd_compress.py

Created June 29, 2023 13:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mikaelhg/dbce60a6454d80085d444058cea1cfad to your computer and use it in GitHub Desktop.
Save mikaelhg/dbce60a6454d80085d444058cea1cfad to your computer and use it in GitHub Desktop.
Download ZIP
Setting zstd compression level to 22 with pandas and parquet
Raw
zstd_compress.py
import pandas as pd
df = ...
df.to_parquet("foobar.parquet.zst", compression='zstd', compression_level=22)
@mikaelhg
Copy link
Author

How to do this shouldn't be this hard to find in documentation.

@mikaelhg
Copy link
Author

https://github.com/apache/arrow/blob/26c25d1b06d3f3740bf954a493c00c090e179abf/python/pyarrow/_parquet.pyx#L1497

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment