$ wget -e robots=off -r -np 'http://example.com/folder/'
- -e robots=off causes it to ignore robots.txt for that domain
- -r makes it recursive
- -np = no parents, so it doesn't follow links up to the parent folder
Created
December 9, 2016 06:38
-
-
Save simonw/27e810771137408fd7834ad153750c41 to your computer and use it in GitHub Desktop.
Recursive wget ignoring robots
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
not fixed