Skip to content

Instantly share code, notes, and snippets.

@miglen
Created July 10, 2024 20:57
Show Gist options
  • Save miglen/12220e618a3c9809443e11790b6cda2d to your computer and use it in GitHub Desktop.
Save miglen/12220e618a3c9809443e11790b6cda2d to your computer and use it in GitHub Desktop.
Clone or mirror website as static with wget
scrapy () {
get \
--mirror \
--no-clobber \
--page-requisites \
--adjust-extension \
--convert-links \
--restrict-file-names=windows \
--domains=$1 \
--no-parent \
-e robots=off \
--execute="robots=off" \
--debug \
http://$1
}
# usage: scrapy domain.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment