Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Sample robots.txt for Magento on Hypernode
User-agent: *
Allow: /
Crawl-delay: 25
sitemap: http://magentotestpakket.nl/sitemap.xml
Disallow: /*?p=
Disallow: /*&p=
Disallow: /*?price=
Disallow: /*&price=
Disallow: /*?color=
Disallow: /*&color=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?dir=
Disallow: /*&dir=
Disallow: /where/
Disallow: /w/
# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /media/captcha/
Disallow: /media/css/
Disallow: /media/css_secure/
Disallow: /media/customer/
Disallow: /media/downloadable/
Disallow: /media/import/
Disallow: /media/js/
Disallow: /media/pdf/
Disallow: /media/sales/
Disallow: /media/tmp/
Disallow: /media/wysiwyg/
Disallow: /media/xmlconnect/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product/gallery/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /get.php
# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?SID=
Disallow: /rss*
Disallow: /*PHPSESSID
User-agent: MJ12bot
Disallow: /
User-agent: Vagabondo
Disallow: /
User-agent: BaiDuSpider
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: BSpider
Disallow: /
@hans2103

This comment has been minimized.

Copy link

hans2103 commented Jan 19, 2016

It is not advisable to block bots to css and js files. Without access to css and js files the search bots cannot see the website as a human being does.

see: https://www.byte.nl/blog/magento-robots-txt
see: https://yoast.com/dont-block-your-css-and-js-files/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.