Skip to content

Instantly share code, notes, and snippets.

@mohammad-ahid
Created January 16, 2017 14:48
Show Gist options
  • Save mohammad-ahid/3e44f60e35b1e0597488cfe39be7fc26 to your computer and use it in GitHub Desktop.
Save mohammad-ahid/3e44f60e35b1e0597488cfe39be7fc26 to your computer and use it in GitHub Desktop.
Just a robots.txt file template which is suitable for WPEngine use
# All Agents
User-agent: *
Disallow: /blogs.dir
Disallow: /cgi-bin
Disallow: /wp-content/cache
Disallow: /wp-content/languages
Disallow: /wp-content/mu-plugins
Disallow: /wp-content/plugins
Disallow: /wp-content/upgrade
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /_wpeprivate
Disallow: /trackback
Disallow: */trackback
Disallow: /*?*
Disallow: /*?
# Google Image
User-agent: Googlebot-Image
Allow: /*
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /trackback
Disallow: /comments
# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /
# digg mirror
User-agent: duggmirror
Disallow: /
Sitemap: www.yourdomain.com/yoursitemap.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment