How To Scrape Images from 4chan Using Wget
This guide is to save other sorry plebs from needing to RTFM in figuring out how to use wget to scrape images from 4chan and other imageboards. There are lots of image downloaders in existence, but they are usually outdated and broken. You will save time following this guide to learn how to use a powerful and general purpose tool instead.
What Is Wget?
Wget is a command-line file downloader that can handle just about any file downloading task normal and power users will ever need to do. It has versions available for Windows, Mac, and Linux. If it is not already installed on your machine, install it now.
wget [options] [urls]
Useful Options for Image Scraping
There are tons more, but these are the most useful ones for this guide.
-rdownloads files recursively, downloading links that are contained in already downloaded documents. This is essential because a common case is one url that contains all image file links.
-l [n]controls the maximum recursion level.
nwill practically always be one for image scraping.
-Hallows downloads from different hosts than the original url. This is useful because many sites show images hosted at different domains.
-D [domains]tells what additional hosts to download from. You will probably have to 'View Source' in your browser to know for sure what to put here.
domainsis a comma separated list of domain names.
-P [prefix directory]tells where to save the downloaded files. The default is the current directory.
-ndavoids creating additional hierarchy.
-A [extensions]tells what file extensions to save.
Putting It Together
To download images from 4chan:
wget -P pictures -nd -r -l 1 -H -D i.4cdn.org -A png,gif,jpg,jpeg,webm [thread-url]
wget -P pictures -nd -r -l 1 -H -D media.8ch.net -A png,gif,jpg,jpeg,webm [thread-url]