Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
WordPress robots.txt Template
# Robots Rule! - Sometimes... #
User-agent: *
Allow: /
# Disallow these directories, url types & file-types
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /cgi-bin
Disallow: /readme.html
Disallow: /license.txt
Disallow: /*?*
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*/wp-*
Disallow: /*/feed/*
Disallow: /*/*?s=*
Disallow: /*/*.js$
Disallow: /*/*.inc$
Allow: /wp-content/uploads/
User-agent: ia_archiver*
Disallow: /
User-agent: duggmirror
Disallow: /
Sitemap: http://example.com/sitemap.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.