Skip to content

Instantly share code, notes, and snippets.

Last active December 2, 2023 02:12
  • Star 35 You must be signed in to star a gist
  • Fork 6 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
Archiving a website with wget

The command I use to archive a single website

wget -mpck --html-extension --user-agent="" -e robots=off --wait 1 -P .

Explanation of the parameters used

  • -m (Mirror) Turns on mirror-friendly settings like infinite recursion depth, timestamps, etc.
  • -c (Continue) Resumes a partially-downloaded transfer
  • -p (Page requisites) Downloads any page dependencies like images, style sheets, etc.
  • -k (Convert) After completing retrieval of all files… converts all absolute links to other downloaded files into relative links converts all relative links to any files that weren’t downloaded into absolute, external links in a nutshell: makes your website archive work locally
  • --html-extension this adds .html after the downloaded filename, to make sure it plays nicely on whatever system you’re going to view the archive on
  • –user-agent=”” Sometimes websites use robots.txt to block certain agents like web crawlers (e.g. GoogleBot) and Wget. This tells Wget to send a blank user-agent, preventing identification. You could alternatively use a web browser’s user-agent and make it look like a web browser, but it probably doesn’t matter.
  • -e robots=off Sometimes you’ll run into a site with a robots.txt that blocks everything. In these cases, this setting will tell Wget to ignore it. Like the user-agent, I usually leave this on for the sake of convenience.
  • –wait 1 Tells Wget to wait 1 second between each action. This will make it a bit less taxing on the servers.
  • -P . set the download directory to something. I left it at the default “.” (which means “here”) but this is where you could pass in a directory path to tell wget to save the archived site. Handy, if you’re doing this on a regular basis (say, as a cron job or something…) http://url-to-site: this is the full URL of the site to download. You’ll likely want to change this.


Copy link

Ham5ter commented Oct 4, 2016

Thank you.

Copy link

Mayank-1234-cmd commented Feb 4, 2021

what i use:

wget --recursive --convert-links -mpck --html-extension --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.146 Safari/537.36." -e robots=off


  • removed the wait, faster
  • useragent makes it look like im using a actual browser (fixes 403 errors)
  • --convert-links converts the links
  • --recursive makes it recursive

Copy link

Question: what is with the limit rate and clobbers in ?
Also this is a good one and

Copy link

encassion commented Nov 18, 2022

This is the code I use to archive a website and be able to feed it into Yacy as a warc file to add to my search results:

wget \ --mirror \ \ --no-verbose \ --warc-cdx \ --page-requisites \ --adjust-extension \ --convert-links \ --no-warc-compression \ --no-warc-keep-log \ --append-output="" \ --execute robots=off \

The main changes are to build a file database and simultaneously create a warc archive of the downloaded site structure. Mainly just so I can confirm the content with my file browser before committing it to Yacy Search. If it isn't worth browsing I'll know immediately and cancel it.

Copy link

Thanks. :)

Copy link

replying to @Mayank-1234-cmd

-m includes the -r flag, which enables recursion.

Copy link

Here is a more explicit/readable version:

wget \
    --user-agent="<user-agent>" \
    --header "cookie: <name>=<value>; <name>=<value>; <etc...>" \
    --adjust-extension \
    --continue \
    --convert-links \
    -e robots=off \
    --no-parent \
    --page-requisites \
    --mirror \

I use --no-parent because I usually do not want anything above the provided URL.

The user-agent for Edge on Windows 10, I believe, if you want to use something generic:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246

If you have to be authenticated to the site, you can use the --header flag to include cookies. If not, it can be removed.

To get necessary cookies, you can use dev tools in your browser. Refresh a page you are already authenticated on while tracking requests and get the cookies from the page request.


Copy link

Copy link

replying to @BrandonKMLee

It does not conflict, to my knowledge; it just includes the recursion flag.

  -m,  --mirror                    shortcut for -N -r -l inf --no-remove-listing

Copy link

mariano-daniel commented Jul 15, 2023

I'm having trouble only wgetting this URL : and all the links that branch out of it, only 1 level away from the link.

When I run wget -e robots=off --recursive -np -k --html-extension ""

it starts downloading all of the forums and all the users. I just want topic?id=3959" to be downloaded, plus the user profiles or links that might branch out from the link, but no deeper than 1 level. Not sure if I'm explaining myself correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment