Skip to content

Instantly share code, notes, and snippets.

@sirodoht

sirodoht/crawl_no200.sh

Created March 12, 2016 20:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sirodoht/3096acb0a15e5ea5fa5e to your computer and use it in GitHub Desktop.
Save sirodoht/3096acb0a15e5ea5fa5e to your computer and use it in GitHub Desktop.
Download ZIP
Crawl your site for anything but HTTP 200's
Raw
crawl_no200.sh
$ wget --spider -r -o ~/crawl_results.log -p http://www.cronweekly.com 2>&1
$ cat ~/crawl_results.log
...
--2016-03-09 22:13:16-- https://www.cronweekly.com/
Reusing existing connection to www.cronweekly.com:443.
HTTP request sent, awaiting response... 200 OK
2016-03-09 22:13:16 (114 MB/s) - 'www.cronweekly.com/index.html' saved [17960]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment