Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Make /robots.txt aware of the Rails environment

Make /robots.txt aware of the Rails environment

You probably don't want Google crawling your development staging app. Here's how to fix that.

$ mv public/robots.txt config/robots.production.txt
$ cp config/robots.production.txt config/robots.development.txt

Now edit config/routes.rb to add a route for /robots.txt, and add the controller code.

def robots
robots = File.read(Rails.root + "config/robots.#{Rails.env}.txt")
render :text => robots, :layout => false, :content_type => "text/plain"
end
# (moved from public/robots.txt)
#
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-Agent: *
Disallow: /
# (moved from public/robots.txt)
#
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /
get '/robots.txt' => 'home#robots'

It cause that all robots request will be handled by rails server not http server?

ok with the cache

robots.development.rb should be renamed robots.development.txt*

Your development staging app should be using the production environment, not development environment. I'd recommend setting a separate environment variable like DISABLE_ROBOTS=true and then using the following instead

class HomeController < ApplicationController
caches_page :robots

def robots
robot_type = ENV["DISABLE_ROBOTS"] == "true" ? "staging" : "production"
robots = File.read(Rails.root + "config/robots/robots.#{robot_type}.txt")
render :text => robots, :layout => false, :content_type => "text/plain"
end

end

In rails 4 you'll need this for page caching https://github.com/rails/actionpack-page_caching

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment