Skip to content

Instantly share code, notes, and snippets.

@underdown

underdown/robots.txt

Created March 4, 2012 23:20
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save underdown/1975339 to your computer and use it in GitHub Desktop.
Save underdown/1975339 to your computer and use it in GitHub Desktop.
Download ZIP
default robots.txt
Raw
robots.txt
User-agent: ia_archiver
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: *
Allow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment