This should be placed in /etc/apache2/conf.d and will provide global robot exclusion for all sites in the server.
The /var/www/robots.txt content is provided.
<DirectoryMatch "/\.git"> | |
Deny from all | |
Satisfy all | |
</DirectoryMatch> | |
Header set X-Robots-Tag "noindex, nofollow, noarchive, nosnippet, noodp, notranslate, noimageindex" | |
<Location "/robots.txt"> | |
SetHandler None | |
</Location> | |
Alias /robots.txt /var/www/robots.txt |
User-agent: * | |
Disallow: / |