raphaelportmann/crawl.sh

Created October 30, 2019 16:17

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/raphaelportmann/662652264dcb154a5d873f80ca230d26.js"></script>
Save raphaelportmann/662652264dcb154a5d873f80ca230d26 to your computer and use it in GitHub Desktop.

Download ZIP

crawl website

Raw

lynx -dump https://example.com | awk '/http/{print $2}' | xargs curl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment