Skip to content

Instantly share code, notes, and snippets.

@theodorocaliari
Last active August 29, 2015 14:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save theodorocaliari/362f263b36fe6fe2f946 to your computer and use it in GitHub Desktop.
Save theodorocaliari/362f263b36fe6fe2f946 to your computer and use it in GitHub Desktop.
# All Agents
User-agent: *
Disallow: /blogs.dir
Disallow: /cgi-bin
Disallow: /wp-content/cache
Disallow: /wp-content/languages
Disallow: /wp-content/mu-plugins
Disallow: /wp-content/plugins
Disallow: /wp-content/upgrade
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /_wpeprivate
Disallow: /trackback
Disallow: */trackback
Disallow: /*?*
Disallow: /*?
# Google Image
User-agent: Googlebot-Image
Allow: /*
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /trackback
Disallow: /comments
# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /
# digg mirror
User-agent: duggmirror
Disallow: /
Sitemap: www.yourdomain.com/yoursitemap.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment