Skip to content

Instantly share code, notes, and snippets.

@CalebMuhia

CalebMuhia/wget.md

Forked from simonw/wget.md
Created March 11, 2019 10:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save CalebMuhia/52d3df876ef154aa547e35adc08680db to your computer and use it in GitHub Desktop.
Save CalebMuhia/52d3df876ef154aa547e35adc08680db to your computer and use it in GitHub Desktop.
Download ZIP
Recursive wget ignoring robots
Raw
wget.md 
$ wget -e robots=off -r -np 'http://example.com/folder/'
  • -e robots=off causes it to ignore robots.txt for that domain
  • -r makes it recursive
  • -np = no parents, so it doesn't follow links up to the parent folder
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment