Skip to content

Instantly share code, notes, and snippets.

@ystoneman
Created June 19, 2022 23:19
Show Gist options
  • Save ystoneman/20978a73b4c2743bc54a184b6c58a2f3 to your computer and use it in GitHub Desktop.
Save ystoneman/20978a73b4c2743bc54a184b6c58a2f3 to your computer and use it in GitHub Desktop.
Loop through ndjson
# Just a note to remember how I looped through rows in an ndjson from S3
from botocore.vendored import requests
import json
import boto3
from io import StringIO
s3 = boto3.resource('s3')
my_bucket = s3.Bucket(bucket)
obj = s3.Object(bucket, key)
response = obj.get()
records = StringIO(response['Body'].read().decode())
k = 0
# ndjson files get loaded as an array of dicts in Python
for row in records:
k += 1
print("k is {} and row is {}".format(k, row))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment