Skip to content

Instantly share code, notes, and snippets.

@herrbischoff

herrbischoff/robot.txt

Last active August 29, 2015 14:18
Show Gist options
  • Save herrbischoff/ba32e2d83e276605fe08 to your computer and use it in GitHub Desktop.
Save herrbischoff/ba32e2d83e276605fe08 to your computer and use it in GitHub Desktop.
Download ZIP
Disallow certain crawlers
Raw
robot.txt
User-agent: SurveyBot
Disallow: /
User-agent: ia_archiver
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment