Skip to content

Instantly share code, notes, and snippets.

@jadedgnome

jadedgnome/spider.sh

Forked from azhawkes/spider.sh
Last active August 29, 2015 14:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jadedgnome/2e45a99d5cd0574a5728 to your computer and use it in GitHub Desktop.
Save jadedgnome/2e45a99d5cd0574a5728 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spider.sh
#!/bin/bash
HOME="http://www.yourdomain.com/some/page"
DOMAINS="yourdomain.com"
DEPTH=2
OUTPUT="./urls.csv"
wget -r --spider --delete-after --force-html -D "$DOMAINS" -l $DEPTH "$HOME" 2>&1 \
| grep '^--' | awk '{ print $3 }' | grep -v '\. \(css\|js\|png\|gif\|jpg\)$' | sort | uniq > $OUTPUT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment