rafaelfoster/wget_ignore_robots.sh

Created May 22, 2018 01:17

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/rafaelfoster/25704e57fd7887167f97ae4a67bddd6c.js"></script>
Save rafaelfoster/25704e57fd7887167f97ae4a67bddd6c to your computer and use it in GitHub Desktop.

Download ZIP

Wget command to ignore robots.txt file

Raw

wget_ignore_robots.sh

wget --random-wait -r -p -e robots=off -U Mozilla

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment