Instantly share code, notes, and snippets.

Embed
What would you like to do?
.htaccess file to disallow access to clients that aren't search engine spiders
# http://www.robotstxt.org/robotstxt.html
# http://httpd.apache.org/docs/2.0/misc/rewriteguide.html
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} !bot
RewriteRule ^robots.txt - [L,R=404]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment