Skip to content

Instantly share code, notes, and snippets.

@xiaodaigh
Created October 20, 2021 12:50
Show Gist options
  • Save xiaodaigh/46e3edad9c72dd10ae415e08ac2953c3 to your computer and use it in GitHub Desktop.
Save xiaodaigh/46e3edad9c72dd10ae415e08ac2953c3 to your computer and use it in GitHub Desktop.
Simple demonstration of writing Parquet to S3
# to run this firstly set up the MinIO server
# on Windows download this https://dl.min.io/server/minio/release/windows-amd64/minio.exe
# download("https://dl.min.io/server/minio/release/windows-amd64/minio.exe")
# to run minio make sure the executable is in the PATH
# then run
# ;minio.exe server /path/to/minio/data
# e.g. minio.exe server c:/minio-data/
using Minio, Parquet, Parquet2, DataFrames, AWSS3
df = DataFrame(a=1:3, b=4:6, c=7:9)
cfg = MinioConfig("http://localhost:9000"; username="admin", password="password")
# using the AWS S3 API
s3_list_buckets(cfg)
p = S3Path("s3://test/plsdel.parquet"; config=cfg)
@time Parquet.write_parquet(p, df)
# read the data back using Parquet2.jl
@time DataFrame(Parquet2.Dataset(p))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment