Skip to content

Instantly share code, notes, and snippets.

@inovarti inovarti/robots.txt Secret forked from deivisonarthur/robots.txt
Last active Aug 29, 2015

Embed
What would you like to do?
User-agent: Googlebot-Image
Allow: /
User-agent: Googlebot
Disallow: /index.php/
Disallow: /*?
Disallow: /checkout/
Disallow: /review/
Disallow: /app/
Disallow: /downloader/
Disallow: /js/
Disallow: /lib/
Disallow: /media/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /var/
Disallow: /catalog/product_compare/
Disallow: /customer/
Disallow: /wishlist/
# Crawlers Setup
User-agent: *
# Directories
Disallow: /index.php/
Disallow: /checkout/
Disallow: /app/
Disallow: /downloader/
Disallow: /js/
Disallow: /lib/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /var/
Disallow: /catalog/product_compare/
Disallow: /customer/
Disallow: /wishlist/
Disallow: /404/
Disallow: /app/
Disallow: /downloader/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
Disallow: /onepagecheckout/
Disallow: /temp/
Disallow: /news/
# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Sitemap: http://www.seushop.com.br/sitemap.xml
User-agent: Baiduspider
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: proximic
Disallow: /
User-agent: Sosospider
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.