Skip to content

Instantly share code, notes, and snippets.

@u0d7i

u0d7i/wget-ignore-robots.txt

Created January 10, 2016 09:46
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save u0d7i/87aa962311f2a7c739aa to your computer and use it in GitHub Desktop.
Save u0d7i/87aa962311f2a7c739aa to your computer and use it in GitHub Desktop.
Download ZIP
Raw
wget-ignore-robots.txt
$ wget -e robots=off --wait 1 http://your.site.here
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment