Skip to content

Instantly share code, notes, and snippets.

@monkut

monkut/iterate_s3_csv.py

Created November 28, 2020 13:24
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save monkut/a74b03da671faeecf97896221dda5e04 to your computer and use it in GitHub Desktop.
Save monkut/a74b03da671faeecf97896221dda5e04 to your computer and use it in GitHub Desktop.
Download ZIP
iterate a csv file in S3 from memory
Raw
iterate_s3_csv.py
import boto3
S3_CLIENT = boto3.client("s3")
def iterate_s3_csv(bucket: str, key: str) -> Generator:
with BytesIO() as bytesin:
S3_CLIENT.download_fileobj(bucket, key, bytesin)
bytesin.seek(0)
stringin = StringIO(bytesin.read().decode("utf8"))
reader = csv.DictReader(stringin)
yield from reader
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment