Skip to content

Instantly share code, notes, and snippets.

@isaacrankin
Last active August 29, 2015 14:16
Show Gist options
  • Save isaacrankin/8176e6bb44b023d349e9 to your computer and use it in GitHub Desktop.
Save isaacrankin/8176e6bb44b023d349e9 to your computer and use it in GitHub Desktop.
Disable robots for staging site
# Use PHP to output robots.txt to avoid accidentally indexing staging sites
RewriteRule ^robots\.txt$ robots.php [NC,L]
<?php
error_reporting(0);
// Default robots - crawl everything
$robotsTxt = "# www.robotstxt.org/
\r\n
# Allow crawling of all content\r\n
User-agent: *";
// If on a staging site prevent indexing, add more domains here if needed
if(strpos($_SERVER['SERVER_NAME'], 'staging.com') !== false){
$robotsTxt = "User-agent: * \r\nDisallow: /";
}else{
// read static robots.txt template
$robotsTxt = @file_get_contents('robots.txt');
}
header('Content-Type:text/plain');
echo $robotsTxt;
# www.robotstxt.org/
# Allow crawling of all content
User-agent: *
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment