Warm up of reverse proxy like Varnish using wget

wget

wget --spider -o wget.log -e robots=off -r -l 5 -p -S -T3 --header="X-Bypass-Cache: 1" -H --domains=example.tld --show-progress www.example.tld

# Options explained
# --spider: Crawl the site
# -o wget.log: Keep the log
# -e robots=off: Ignore robots.txt
# -r: specify recursive download
# -l 5: Depth to search. I.e 1 means 'crawl the homepages'.  2 means 'crawl the homepage and all pages it links to'...
# -p: get all images, etc. needed to display HTML page
# -S: print server response (to the log)
# --delete-after - delete the file once it os down loaded (we are only warming caches after all not mirroring the site)
# -T 3 Timeout after 3 seconds - default is 900
# --header="X-Bypass-Cache: 1": Set a header (this one bypasses Varnish cache). You might need user agent like
# --header="User-Agent:CacheWarmer"
# or use real one
# --header="User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36"
# multiple --headers can be provided
# --show-progress - list the urls we are warming
# -H Span domains (only used so we can limit with -D/--domains)
# --domains=example.tld - will warm up subdomains too; requires -H
# --max-redirect=0 - don't follow redirects (not in example above but useful)
# --limit-rate=20k - if you don't want to hammer your site
# www.example.tld: URL to start crawling