Skip to content

Instantly share code, notes, and snippets.

@robcowie

robcowie/list_s3_with_metadata.py

Created January 28, 2019 12:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save robcowie/7391ea2781b593cc03455b72676c9d80 to your computer and use it in GitHub Desktop.
Save robcowie/7391ea2781b593cc03455b72676c9d80 to your computer and use it in GitHub Desktop.
Download ZIP
List S3 with pagination and metadata
Raw
list_s3_with_metadata.py
def list_s3_with_metadata(s3_conn, prefix):
"""List all keys at `prefix` and return metadata."""
bucket, prefix = prefix.split('://')[1].split('/', 1)
paginator = s3_conn.get_paginator('list_objects_v2')
response = paginator.paginate(Bucket=bucket, Prefix=prefix)
def attrs(d):
return {'Key': 's3://{}/{}'.format(bucket, d['Key']), 'ETag': d['ETag'].replace('"', ''), 'Size': d['Size']}
return (attrs(k) for page in response for k in page['Contents'])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment