Skip to content

Instantly share code, notes, and snippets.

@MicFin
Forked from Integralist/Website Crawler.sh
Created May 5, 2017 04:22
Show Gist options
  • Save MicFin/b453d7a7d4fe7d344883129661f709cb to your computer and use it in GitHub Desktop.
Save MicFin/b453d7a7d4fe7d344883129661f709cb to your computer and use it in GitHub Desktop.
Website Crawler
wget --mirror \
--convert-links \
--adjust-extension \
--page-requisites \
--header="Host: www.example.com" \
--no-parent https://beepboop.cloudfront.net
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment