Skip to content

Instantly share code, notes, and snippets.

@parhamr

parhamr/wget-url-cleanup.md

Created October 31, 2014 21:33
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save parhamr/358a90b3bdbc955d4f10 to your computer and use it in GitHub Desktop.
Save parhamr/358a90b3bdbc955d4f10 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
wget-url-cleanup.md

Use the log output from wget to capture a list of URLs:

wget -a wget-log.txt --ignore-tags=img,link,script -X admin -e robots=off -nv --header="Accept: text/html" -r -l 0 --domains=example.org --no-cookies http://example.org/

Process that log file to collect a sorted list of unique URLs:

cat wget-log.txt | grep -F 'URL:http' | awk '{print $3}' | sort | uniq | sed -e s/^URL:// > tmp.txt

…and then use that file with Siege for various tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment