Skip to content

Instantly share code, notes, and snippets.

@sirodoht

sirodoht/crawl_404.sh

Created March 12, 2016 20:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sirodoht/31ff5dea5a468d2f3701 to your computer and use it in GitHub Desktop.
Save sirodoht/31ff5dea5a468d2f3701 to your computer and use it in GitHub Desktop.
Download ZIP
Crawl your site for 404's
Raw
crawl_404.sh
$ wget --spider -r -p http://www.cronweekly.com 2>&1 | grep -B 2 ' 404 '
--2016-03-09 22:08:14-- https://www.cronweekly.com/three-tiers-package-managers/
Reusing existing connection to www.cronweekly.com:443.
HTTP request sent, awaiting response... 404 Not Found
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment