Skip to content

Instantly share code, notes, and snippets.

@wpperform
Last active Jan 27, 2016
Embed
What would you like to do?
wpPERFORM.com's recommended robots.txt file
User-agent: *
Allow: /*.js*
Allow: /*.css*
Disallow: /*?
Disallow: /feed/
Disallow: */feed/
Disallow: /comments/
Disallow: */comments/
Disallow: /trackback/
Disallow: */trackback/
Disallow: /wp-content/upgrade/
# WP admin directories
# GoogleBot needs access to js & css, and Google's order of precedence blocks them if they are disallowed here
#Disallow: /wp-admin/
#Disallow: /wp-includes/
#Disallow: /wp-content/themes/
#Disallow: /wp-content/plugins/
#Disallow: /wp-content/mu-plugins/
# if you use NextGen Gallery, uncomment the next line and replace {siteID} with your numeric site ID
#Disallow: /wp-content/uploads/sites/{siteID}/gallery/cache
# if you use the WP-Print plugin, uncomment the next 2 lines
#Disallow: /print/
#Disallow: */print/
Allow: /
# if you want to block the Internet Archive crawler, https://archive.org/details/archive.org_bot, uncomment the next 2 lines
#User-agent: archive.org_bot*
#Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment