Skip to content

Instantly share code, notes, and snippets.

@SethosII
Last active November 16, 2019 20:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save SethosII/20db363aef55d3252dd1c47c960e16a2 to your computer and use it in GitHub Desktop.
Save SethosII/20db363aef55d3252dd1c47c960e16a2 to your computer and use it in GitHub Desktop.
Download all of the xkcd comics
#!/bin/bash
TOTAL=$(curl https://xkcd.com/info.0.json 2> /dev/null | grep -Po '"num": [0-9]+' | grep -Po '[0-9]+')
# download JSON and image file
echo "downloading ..."
for i in $(seq 1 $TOTAL)
do
echo -ne "\r$i/$TOTAL"
URL=https://xkcd.com/$i/info.0.json
JSON=$(curl $URL 2> /dev/null)
IMG=$(echo $JSON | grep -Po '"img":\s"https:\/\/imgs.xkcd.com\/comics\/[\w\(\)-]+.(png|jpg|gif)"' | grep -Po 'https://imgs.xkcd.com\/comics\/[\w\(\)-]+.(png|jpg|gif)' | sed 's/\\//g')
TYPE=$(echo $IMG | grep -o "...$")
NUM=$(echo $JSON | grep -Po '"num": [0-9]+' | grep -Po '[0-9]+')
echo $JSON > "$NUM.json"
wget $IMG -O "$NUM.$TYPE" 2> /dev/null
done
echo
# verification, there should be 2 files for each comic (JSON, image)
echo "verifiying ... (some don't have an image (404, 1446, 1608, 1663), so a small amount of errors being reported is ok)"
for i in $(seq 1 $TOTAL)
do
echo -ne "\r$i/$TOTAL"
if [[ $(ls $i.* 2> /dev/null | wc -l) -lt 2 ]]
then
echo -e "\rerror at $i"
fi
done
echo
exit 0
@dikiaap
Copy link

dikiaap commented Apr 28, 2018

I think by combining your script with @WJH's script will make the file name more detailed and easy to remember.
From 1434.jpg and 1434.json to xkcd 1434: Where Do Birds Go.jpg and xkcd 1434: Where Do Birds Go.json

https://gist.github.com/dikiaap/96b9725f8c55d52b603d773de90fd3dd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment