Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
.htaccess file to disallow access to clients that aren't search engine spiders
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} !bot
RewriteRule ^robots.txt - [L,R=404]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment