Skip to content

Instantly share code, notes, and snippets.

@edsu

edsu/craigslist-crawl

Created August 7, 2012 16:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save edsu/3286880 to your computer and use it in GitHub Desktop.
Save edsu/3286880 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
craigslist-crawl
Python 2.7.3 (default, Apr 20 2012, 22:44:07)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url("http://washingtondc.craigslist.org/robots.txt")
>>> rp.read()
>>> rp.can_fetch("*", "http://washingtondc.craigslist.org/doc/prk/3189693671.html")
True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment