Skip to content

Instantly share code, notes, and snippets.

@j2labs
Created November 21, 2014 15:20
Show Gist options
  • Save j2labs/9f2412b930f8eeed36ef to your computer and use it in GitHub Desktop.
Save j2labs/9f2412b930f8eeed36ef to your computer and use it in GitHub Desktop.
Simple python spider
#!/usr/bin/env python
### To use, first install gevent and then requests
###
### $ pip install gevent requests
from gevent import monkey; monkey.patch_all() ### gevent first
import gevent
import json
import requests
def handler(url):
"""Scrapes a single URL"""
response = requests.get(url)
print '%s: %s' % (response.status_code, url)
return response
greenlets = []
for url in ['http://google.com', 'http://yahoo.com']:
greenlets.append(gevent.spawn(handler, url))
gevent.joinall(greenlets)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment