Skip to content

Instantly share code, notes, and snippets.

@thegulshankumar thegulshankumar/robots.txt
Last active Dec 12, 2017

Embed
What would you like to do?
Idle robots.txt for WordPress
Sitemap: https://www.example.com/sitemap_index.xml
User-agent: *
Disallow: /?s=*
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-admin/images/
Allow: /wp-admin/css/
Allow: /wp-admin/js/
@thegulshankumar

This comment has been minimized.

Copy link
Owner Author

commented Apr 5, 2017

Line by line Explanation

  1. Sitemap Declared
  2. For all user-agents
  3. Blocked Search Results
  4. Blocked administrative path
  5. Allowed ajax theme elements
  6. Allowed wp-admin images path for Embed content
  7. Allowed wp-admin CSS
  8. Allowed wp-admin JS

[The robots.txt rule is written as per Google Mobile Friendly guidelines which says, do not block CSS and JS path.]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.