Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
robots.txt for TYPO3
User-agent: *
# Only allow URLs generated with RealURL
Disallow: /*?id=*
Disallow: /*&id=*
# L=0 is the default language
Disallow: /*?L=0*
Disallow: /*&L=0*
# typeNum = 98 is usually the print version.
Disallow: /*?type=98*
Disallow: /*&type=98*
# Should always be protected (.htaccess)
Disallow: /*/Private/*
Disallow: /fileadmin/templates/html/*
Disallow: /*/Configuration/*
Disallow: /typo3temp/*
Allow: /typo3temp/*.css$
Allow: /typo3temp/*.css.*.gzip$
Allow: /typo3temp/*.js$
Allow: /typo3temp/*.js.*.gzip$
Allow: /typo3temp/*.jpg$
Allow: /typo3temp/*.gif$
Allow: /typo3temp/*.png$
Disallow: *.sql
Disallow: *.sql.gz
Sitemap: http://www.example.com/index.php?eID=dd_googlesitemap
@josefglatz

This comment has been minimized.

Copy link

commented Sep 30, 2016

What about

Disallow: /*?tx_indexedsearch

or solr search query

@stat1x

This comment has been minimized.

Copy link

commented Jan 20, 2017

This:

Disallow: /typo3temp/*
Allow: /typo3temp/*.css$
Allow: /typo3temp/*.css.*.gzip$
Allow: /typo3temp/*.js$
Allow: /typo3temp/*.js.*.gzip$
Allow: /typo3temp/*.jpg$
Allow: /typo3temp/*.gif$
Allow: /typo3temp/*.png$

Should be,

Disallow: /typo3temp/*
Allow: /typo3temp/*.css
Allow: /typo3temp/*.css.*.gzip
Allow: /typo3temp/*.js
Allow: /typo3temp/*.js.*.gzip
Allow: /typo3temp/*.jpg
Allow: /typo3temp/*.gif
Allow: /typo3temp/*.png

To work

@mcmz

This comment has been minimized.

Copy link

commented Jun 9, 2018

Isn't
Disallow: /typo3conf/
worth to be mentioned?

@sebkln

This comment has been minimized.

Copy link

commented Nov 22, 2018

This:

Disallow: /typo3temp/*
Allow: /typo3temp/*.css$
Allow: /typo3temp/*.css.*.gzip$
Allow: /typo3temp/*.js$
Allow: /typo3temp/*.js.*.gzip$
Allow: /typo3temp/*.jpg$
Allow: /typo3temp/*.gif$
Allow: /typo3temp/*.png$

Should be,

Disallow: /typo3temp/*
Allow: /typo3temp/*.css
Allow: /typo3temp/*.css.*.gzip
Allow: /typo3temp/*.js
Allow: /typo3temp/*.js.*.gzip
Allow: /typo3temp/*.jpg
Allow: /typo3temp/*.gif
Allow: /typo3temp/*.png

To work

Thanks. This change allows Google to crawl stylesheets and JavaScripts and prevents error messages in the Google Search Console.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.