Skip to content

Instantly share code, notes, and snippets.

@brownman
Forked from cloudsben/gist:4490135
Last active August 29, 2015 14:02
Show Gist options
  • Save brownman/d2c2905fb24b8591a7cc to your computer and use it in GitHub Desktop.
Save brownman/d2c2905fb24b8591a7cc to your computer and use it in GitHub Desktop.
download_entire_website.sh
# -p parameter tells wget to include all files, including images.
# -e robots=off you don't want wget to obey by the robots.txt file
# -U mozilla as your browsers identity.
# --random-wait to let wget chose a random number of seconds to wait, avoid get into black list.
# Other Useful wget Parameters:
# --limit-rate=20k limits the rate at which it downloads files.
# -b continues wget after logging out.
# -o $HOME/wget_log.txt logs the output
wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment