Skip to content

Instantly share code, notes, and snippets.

@klongdesigns

klongdesigns/Default Robots.txt file for Wordpress

Created April 27, 2014 04:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save klongdesigns/11337892 to your computer and use it in GitHub Desktop.
Save klongdesigns/11337892 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Default Robots.txt file for Wordpress
#### Robots.txt file to block others
User-agent: MJ12bot*
Disallow: /
User-agent: NinjaBot
Allow: /
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: /wp-content/plugins/
Crawl-delay: 30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment