Skip to content

Instantly share code, notes, and snippets.

@sbalnojan

sbalnojan/dump_df_to_s3fs.py

Created May 22, 2019 12:02
Show Gist options
  • Save sbalnojan/87e960b05708d6af4f55fdd4ed563314 to your computer and use it in GitHub Desktop.
Save sbalnojan/87e960b05708d6af4f55fdd4ed563314 to your computer and use it in GitHub Desktop.
Download ZIP
dump a df to s3 via s3fs
Raw
dump_df_to_s3fs.py
# ...
# your data preprocessing...
if __name__ == "__main__":
# ...
# your data preprocessing...
s3 = s3fs.S3FileSystem(anon=False)
df = pd.DataFrame(data={"foo":[0]})
print(df)
with s3.open(f"{BUCKET_NAME}/data_bow_limit40000.csv",'w') as f:
df.to_csv(f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment