Skip to content

Instantly share code, notes, and snippets.

shahidash

  • applied informatics
  • srinagar
Block or report user

Report or block shahidash

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@uhho
uhho / pandas_s3_streaming.py
Last active Dec 14, 2019
Streaming pandas DataFrame to/from S3 with on-the-fly processing and GZIP compression
View pandas_s3_streaming.py
def s3_to_pandas(client, bucket, key, header=None):
# get key using boto3 client
obj = client.get_object(Bucket=bucket, Key=key)
gz = gzip.GzipFile(fileobj=obj['Body'])
# load stream directly to DF
return pd.read_csv(gz, header=header, dtype=str)
def s3_to_pandas_with_processing(client, bucket, key, header=None):
You can’t perform that action at this time.