Skip to content

Instantly share code, notes, and snippets.

@smajda
Created April 15, 2013 15:01
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save smajda/5388745 to your computer and use it in GitHub Desktop.
Save smajda/5388745 to your computer and use it in GitHub Desktop.
wget mirror with options I have to google for everytime.
#!/bin/sh
url='http://someurl.com/'
wait=2 # wait between each request
cutdirs=2 # i.e. if '/foo/bar/data' -> start at 'data'
wget \
--execute="robots=off" \
--mirror \
--convert-links \
--no-parent \
--no-host \
--cut-dirs=$cutdirs \
--wait=$wait \
${url}
# Other handy options not used here:
# --reject index.html* # if you don't actually want index.htmls
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment