Skip to content

Instantly share code, notes, and snippets.

@NorthIsUp
Created August 27, 2013 20:48
Show Gist options
  • Save NorthIsUp/6358981 to your computer and use it in GitHub Desktop.
Save NorthIsUp/6358981 to your computer and use it in GitHub Desktop.
def fetch(self, url):
logging.info('Fetching %s', url)
urlparts = urlparse(url)
conn = httplib.HTTPConnection(urlparts.hostname, urlparts.port, timeout=30)
try:
conn.request('GET', urlparts.path + '?' + urlparts.query)
resp = conn.getresponse()
self.results[url] = resp.read()
except Exception:
logging.error('Failed to fetch %s', url, exc_info=True)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment