Skip to content

Instantly share code, notes, and snippets.

@benmezger

benmezger/gist:5130523

Created March 10, 2013 21:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save benmezger/5130523 to your computer and use it in GitHub Desktop.
Save benmezger/5130523 to your computer and use it in GitHub Desktop.
Download ZIP
Perhaps a robotparser module bug?
Raw
gistfile1.py
In [65]: rp.set_url("https://www.facebook.com/robots.txt")
In [66]: rp.can_fetch("Hello_crawler", "http://www.facebook.com")
Out[66]: True
In [67]: rp.can_fetch("Hello_crawler", "http://www.facebook.com/sharer/")
Out[67]: True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment