Skip to content

Instantly share code, notes, and snippets.

@bbayles
Created July 14, 2020 20:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bbayles/b5e377a7626b544d83f03aeb877329c8 to your computer and use it in GitHub Desktop.
Save bbayles/b5e377a7626b544d83f03aeb877329c8 to your computer and use it in GitHub Desktop.
from gzip import open as gz_open
from boto3 import client as boto3_client
def get_lengths(bucket, key, open_func=gz_open, s3_client=None, chunk_size=8192):
s3_client = s3_client or boto3_client('s3')
resp = s3_client.get_object(Bucket=bucket, Key=key)
uncompressed_length = 0
with open_func(resp['Body'], mode='rb') as f:
for chunk in iter(lambda: f.read(chunk_size), b''):
uncompressed_length += len(chunk)
return resp['ContentLength'], uncompressed_length
get_lengths(bucket, key)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment