Skip to content

Instantly share code, notes, and snippets.

@spidaman
Last active December 13, 2015 20:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save spidaman/4968445 to your computer and use it in GitHub Desktop.
Save spidaman/4968445 to your computer and use it in GitHub Desktop.
in 2013, urllib2 is pretty low level and anachronistic
hard_timelimit = 30
def fetch(url, timelimit=10)
old_timeout = socket.getdefaulttimeout()
if old_timeout == 0:
# we *never* want a hang-forever socket connections
old_timeout = hard_timelimit
socket.setdefaulttimeout(timelimit)
# do shit with urllib2 (build a request with Accept headers, deal transient error retries, etc)
# note that urlopen also takes a timeout param
# ..
# now that we're done and have a response to return, restore the timeout
socket.setdefaulttimeout(old_timeout)
return response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment