Skip to content

Instantly share code, notes, and snippets.

@chankruze
Forked from xflr6/urlretrieve.py
Created June 25, 2021 19:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save chankruze/ba55bd23bd1a951e4ecb3cecfffe8e52 to your computer and use it in GitHub Desktop.
Save chankruze/ba55bd23bd1a951e4ecb3cecfffe8e52 to your computer and use it in GitHub Desktop.
Replacement for urllib.urlretrieve(url, filename) using the requests library
# urlretrieve.py - implement urllib.urlretrieve(url, filename) with requests
import contextlib
import urllib
import requests
def urlretrieve(url, filename):
with contextlib.closing(requests.get(url, stream=True)) as r:
r.raise_for_status()
with open(filename, 'wb') as f:
for chunk in r.iter_content(chunk_size=8 * 1024):
f.write(chunk)
return filename, r.headers
def urlretrieve(url, filename):
with contextlib.closing(requests.get(url, stream=True)) as r:
r.raise_for_status()
size = int(r.headers.get('Content-Length', '-1'))
read = 0
with open(filename, 'wb') as f:
for chunk in r.iter_content(chunk_size=8 * 1024):
read += len(chunk)
f.write(chunk)
if size >= 0 and read < size:
msg = 'retrieval incomplete: got only %i out of %i bytes' % (read, size)
raise urllib.ContentTooShortError(msg, (filename, r.headers))
return filename, r.headers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment