Skip to content

Instantly share code, notes, and snippets.

@bojanbjelic

bojanbjelic/spider-urls.sh

Created February 14, 2013 10:37
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bojanbjelic/4951903 to your computer and use it in GitHub Desktop.
Save bojanbjelic/4951903 to your computer and use it in GitHub Desktop.
Download ZIP
spider unique URLs on pages, starting from one URL
Raw
spider-urls.sh
wget --spider --force-html -r -l1 http://tigar.ir.ttweb.net/ 2>&1 | grep '^--' | awk '{ print $3 }' | grep -v '\.\(css\|txt\|png\|gif\|jpg\|js\)$' | sort --unique
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment