Create a gist now

Instantly share code, notes, and snippets.

Python 2.7.3 (default, Apr 20 2012, 22:44:07)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url("http://washingtondc.craigslist.org/robots.txt")
>>> rp.read()
>>> rp.can_fetch("*", "http://washingtondc.craigslist.org/doc/prk/3189693671.html")
True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment