Skip to content

Instantly share code, notes, and snippets.

View danielrmeyer's full-sized avatar

Daniel Meyer danielrmeyer

View GitHub Profile
@viesti
viesti / utils.py
Created May 2, 2019 15:54
Stream and decompress a gzip file from S3
import boto3
import zlib
def decompress_object(bucket, key, out_file):
s3 = boto3.client("s3")
body = s3.get_object(Bucket=bucket, Key=key)['Body']
with open(out_file, "ab") as out:
decompressor = zlib.decompressobj(zlib.MAX_WBITS|32)
for chunk in body.iter_chunks():
out.write(decompressor.decompress(chunk))