Skip to content

Instantly share code, notes, and snippets.

@jezhalford

jezhalford/trawl.sh

Created April 16, 2015 13:25
Show Gist options
  • Save jezhalford/41d8ad4c79ff03c72a73 to your computer and use it in GitHub Desktop.
Save jezhalford/41d8ad4c79ff03c72a73 to your computer and use it in GitHub Desktop.
Download ZIP
Trawl for broken links
Raw
trawl.sh
wget --spider -o ./wget.log -r -p http://www.example.com
grep -B 2 '\.\.\. 40[0-9]\|\.\.\. 50[0-9]' wget.log | sed '/Reusing/d' | sed 's/^HTTP.*\.\.\. //' | sed 's/--.*-- //' | sed '/^--$/d' > broken-links.log
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment