Skip to content

Instantly share code, notes, and snippets.

@tjkelly
Last active January 26, 2023 19:54
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save tjkelly/736fe21232dc1701d24c702db4eecd08 to your computer and use it in GitHub Desktop.
WordPress URL Exclude List for Screaming Frog Spider
/**
* WordPress URL Exclude List for Screaming Frog Spider
* @author TJ Kelly - https://tjkelly.com
* @desc Full article — https://tjkelly.com/blog/screaming-frog-exclude-wordpress/
* @date 2021-07-08
*/
https://example.com/wp-content/.*
https://example.com/wp-content/mu-plugins/.*
https://example.com/wp-content/plugins/.*
https://example.com/wp-content/themes/.*
https://example.com/wp-content/upgrade/.*
https://example.com/wp-content/uploads/.*
https://example.com/wp-includes/.*
https://example.com/wp-admin/.*
https://example.com/index.php
https://example.com/license.txt
https://example.com/readme.html
https://example.com/wp-activate.php
https://example.com/wp-blog-header.php
https://example.com/wp-comments-post.php
https://example.com/wp-config.php
https://example.com/wp-config-sample.php
https://example.com/wp-cron.php
https://example.com/wp-links-opml.php
https://example.com/wp-load.php
https://example.com/wp-login.php
https://example.com/wp-mail.php
https://example.com/wp-settings.php
https://example.com/wp-signup.php
https://example.com/wp-trackback.php
https://example.com/xmlrpc.php
https://example.com/author/.*
https://example.com/category/.*
https://example.com/tag/.*
https://example.com/page/2/.*
https://example.com/page/3/.* — etc.
https://example.com/bin/.*
https://example.com/boot/.*
https://example.com/cdn-cgi/.*
https://example.com/cgi-bin/.*
https://example.com/dev/.*
https://example.com/etc/.*
https://example.com/home/.*
https://example.com/lib/.*
https://example.com/media/.*
https://example.com/mnt/.*
https://example.com/opt/.*
https://example.com/run/.*
https://example.com/sbin/.*
https://example.com/srv/.*
https://example.com/tmp/.*
https://example.com/usr/.*
https://example.com/var/.*
https://example.com/en/.*
https://example.com/es/.*
https://example.com/fr/.*
https://en.example.com/.*
https://es.example.com/.*
https://fr.example.com/.*
https://blog.example.com/.*
https://forum.example.com/.*
https://info.example.com/.*
https://shop.example.com/.*
https://store.example.com/.*
https://clickfunnels.example.com/.*
https://eloqua.example.com/.*
https://hubspot.example.com/.*
https://instapage.example.com/.*
https://kajabi.example.com/.*
https://leadpages.example.com/.*
https://marketo.example.com/.*
https://unbounce.example.com/.*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment