Skip to content

Instantly share code, notes, and snippets.

@lolosssss
Created September 30, 2018 07:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lolosssss/ae5dc6efc34b860294d6322c24a8a1ec to your computer and use it in GitHub Desktop.
Save lolosssss/ae5dc6efc34b860294d6322c24a8a1ec to your computer and use it in GitHub Desktop.
Downloading a web page with 5XX retry
import urllib.request
from urllib.error import URLError, HTTPError, ContentTooShortError
def download(url, num_retries=2):
print('Downloading:', url)
try:
html = urllib.request.urlopen(url).read()
except (URLError, HTTPError, ContentTooShortError) as e:
print('Download error:', e.reason)
html = None
if num_retries > 0:
if (hasattr(e, 'code') and 500 <= e.code < 600):
return download(url, num_retries - 1)
return html
if __name__ == '__main__':
html_404 = download('http://httpstat.us/404')
print(html_404)
html_500 = download('http://httpstat.us/500')
print(html_500)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment