Skip to content

Instantly share code, notes, and snippets.

@amalgjose
Last active September 10, 2020 03:47
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Embed
What would you like to do?
Sample program to stream large data from a REST or HTTP endpoint using Python. For more details refer to https://amalgjose.com/2020/08/10/program-to-stream-large-data-from-a-rest-endpoint-using-python/
import requests
session = requests.Session()
authentication = {"USER":"", "PASSWORD":""}
payload = {"query":"some query"}
local_file = "data.json"
# This is a dummy URL. You can replace this with the actual URL
URL = "https://sampledatadowload.com/somedata"
# This is a POST request
with session.post(URL, stream=True, data=payload, auth=(authentication["USER"], authentication["PASSWORD"]), verify=False) as r:
r.raise_for_status()
with open(local_file, 'wb') as f:
for chunk in r.iter_content(chunk_size=128):
f.write(chunk)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment