Skip to content

Instantly share code, notes, and snippets.

@gr1zix
Created October 30, 2023 15:02
Show Gist options
  • Save gr1zix/9b9fa96550560c2a01b5dc8d9ec27232 to your computer and use it in GitHub Desktop.
Save gr1zix/9b9fa96550560c2a01b5dc8d9ec27232 to your computer and use it in GitHub Desktop.
Wordpress robots.txt basic
User-agent: *
Disallow: /cgi-bin
Disallow: /?
Disallow: /admin/
Disallow: /login/
Disallow: /wp-
Disallow: *?p=
Disallow: *?s=
Disallow: *&s=
Disallow: /search/
Disallow: /tag/
Disallow: /archive/
Disallow: /author/
Disallow: /users/
Disallow: */trackback
Disallow: */feed/
Disallow: */rss/
Disallow: */embed
Disallow: */wlwmanifest.xml
Disallow: /xmlrpc.php
Disallow: *utm=
Disallow: *openstat=
Disallow: /*?reviews-page=*
Disallow: /*?*reviews-page=*
Disallow: /*?*replytocom=*
Allow: */uploads
Allow: /*/*.js
Allow: /*/*.css
Allow: /wp-*.png
Allow: /wp-*.svg
Allow: /wp-*.woff
Allow: /wp-*.ttf
Allow: /wp-*.jpg
Allow: /wp-*.jpeg
Allow: /wp-*.gif
Allow: /wp-admin/admin-ajax.php
Clean-Param: utm_source&utm_medium&utm_campaign
Clean-Param: openstat
Sitemap: https://domain.com/sitemap.xml
Host: https://domain.com/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment