Last active
January 27, 2016 15:13
-
-
Save wpperform/6043696 to your computer and use it in GitHub Desktop.
wpPERFORM.com's recommended robots.txt file
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
User-agent: * | |
Allow: /*.js* | |
Allow: /*.css* | |
Disallow: /*? | |
Disallow: /feed/ | |
Disallow: */feed/ | |
Disallow: /comments/ | |
Disallow: */comments/ | |
Disallow: /trackback/ | |
Disallow: */trackback/ | |
Disallow: /wp-content/upgrade/ | |
# WP admin directories | |
# GoogleBot needs access to js & css, and Google's order of precedence blocks them if they are disallowed here | |
#Disallow: /wp-admin/ | |
#Disallow: /wp-includes/ | |
#Disallow: /wp-content/themes/ | |
#Disallow: /wp-content/plugins/ | |
#Disallow: /wp-content/mu-plugins/ | |
# if you use NextGen Gallery, uncomment the next line and replace {siteID} with your numeric site ID | |
#Disallow: /wp-content/uploads/sites/{siteID}/gallery/cache | |
# if you use the WP-Print plugin, uncomment the next 2 lines | |
#Disallow: /print/ | |
#Disallow: */print/ | |
Allow: / | |
# if you want to block the Internet Archive crawler, https://archive.org/details/archive.org_bot, uncomment the next 2 lines | |
#User-agent: archive.org_bot* | |
#Disallow: / |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment