Make /robots.txt aware of the Rails environment
You probably don't want Google crawling your development staging app. Here's how to fix that.
$ mv public/robots.txt config/robots.production.txt $ cp config/robots.production.txt config/robots.development.txt
config/routes.rb to add a route for
/robots.txt, and add the controller code.
It cause that all robots request will be handled by rails server not http server?
ok with the cache