Skip to content

Instantly share code, notes, and snippets.

View thewhitegrizzli's full-sized avatar
🐼
just bearing around

aiav thewhitegrizzli

🐼
just bearing around
  • Ubud, Bali
View GitHub Profile
@simonw
simonw / wget.md
Created December 9, 2016 06:38
Recursive wget ignoring robots
$ wget -e robots=off -r -np 'http://example.com/folder/'
  • -e robots=off causes it to ignore robots.txt for that domain
  • -r makes it recursive
  • -np = no parents, so it doesn't follow links up to the parent folder