Skip to content

Instantly share code, notes, and snippets.

@benmezger

benmezger/gist:5135315

Created March 11, 2013 16:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save benmezger/5135315 to your computer and use it in GitHub Desktop.
Save benmezger/5135315 to your computer and use it in GitHub Desktop.
Download ZIP
robotparser failing?
Raw
gistfile1.py
In [2]: import robotparser
In [3]: x = robotparser.
robotparser.Entry robotparser.RuleLine robotparser.urllib
robotparser.RobotFileParser robotparser.URLopener robotparser.urlparse
In [3]: x = robotparser.RobotFileParser()
In [4]: x.set_url("http://www.google.com/robots.txt")
In [5]: x.read()
In [6]: x.can_fetch("My_Crawler", "/catalogs")
Out[6]: False
In [7]: x.can_fetch("Hello", "/catalogs")
Out[7]: False
In [8]: x.can_fetch("*", "/catalogs")
Out[8]: False
In [9]: x.can_fetch("My_Crawler", "/catalogs/p?")
Out[9]: False
In [10]: x.can_fetch("*", "/catalogs/p?")
Out[10]: False
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment