Skip to content

Instantly share code, notes, and snippets.

@jadedgnome
Forked from fedecarg/wget-spider.md
Last active August 29, 2015 14:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jadedgnome/99cd4a671abbde1eeb52 to your computer and use it in GitHub Desktop.
Save jadedgnome/99cd4a671abbde1eeb52 to your computer and use it in GitHub Desktop.

Extract links from a BBC responsive site

DOMAIN="m.bbc.co.uk"
SERVICE="hindi"
HTTP_USER_AGENT="Mozilla/5.0 (iPhone; Mobile; AppleWebKit; Safari)"
EXCLUDE_EXTENSIONS="\.\(txt\|css\|js\|png\|gif\|jpg\)$"
MAX_DEPTH="3"

wget --spider --no-directories --no-parent --force-html --recursive \
 --level=$MAX_DEPTH --no-clobber \
 --domains=$DOMAIN --include-directories=$SERVICE \
 --user-agent="$HTTP_USER_AGENT" http://$DOMAIN/$SERVICE 2>&1 \
 | grep '^--' \
 | awk '{ print $3 }' \
 | grep -v $EXCLUDE_EXTENSIONS \
 | sort | uniq > $SERVICE.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment