Skip to content

Instantly share code, notes, and snippets.

@privatezero

privatezero/warc.sh

Last active February 28, 2019 20:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save privatezero/f20e690f44e9deb7b70c9eb7df6d1736 to your computer and use it in GitHub Desktop.
Save privatezero/f20e690f44e9deb7b70c9eb7df6d1736 to your computer and use it in GitHub Desktop.
Download ZIP
university homepage 2 warc
Raw
warc.sh
wget -r -l 2 -p -k -H --tries=2 --timeout=5 -D 'wsu.edu' "http://wsu.edu" --warc-file="$(date +%Y-%m-%d)-wsu-homepage" --warc-cdx
# Additions for CSS etc developers.google.com,access.wsu.edu,policies.wsu.edu,portal.wsu.edu,brand.wsu.edu,s3.wp.wsu.edu,s2.googleusercontent.com,repo.wsu.edu,ucomm.wsu.edu,fonts.googleapis.com,fonts.gstatic.com,googletagmanager.com,ajax.googleapis.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment