Skip to content

Instantly share code, notes, and snippets.

@korniichuk
Created August 19, 2019 15:47
Show Gist options
  • Save korniichuk/a21568649c231c412695ccde617c3403 to your computer and use it in GitHub Desktop.
Save korniichuk/a21568649c231c412695ccde617c3403 to your computer and use it in GitHub Desktop.
s3fs with gzip and pandas
# Version: 0.1a1
import gzip
import pandas as pd
import s3fs
src = 's3://bucket/file.gz'
fs = s3fs.S3FileSystem(anon=False)
with fs.open(src, 'rb') as f:
g = gzip.GzipFile(fileobj=f)
df = pd.read_json(g)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment