Created

Embed URL

HTTPS clone URL

SSH clone URL

You can clone with HTTPS or SSH.

Download Gist
View craigslist-crawl
1 2 3 4 5 6 7 8 9
Python 2.7.3 (default, Apr 20 2012, 22:44:07)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url("http://washingtondc.craigslist.org/robots.txt")
>>> rp.read()
>>> rp.can_fetch("*", "http://washingtondc.craigslist.org/doc/prk/3189693671.html")
True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.