Skip to content

Instantly share code, notes, and snippets.

@jgusta
Last active September 2, 2021 14:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jgusta/b683e8224e445b54ab21ead7081a088d to your computer and use it in GitHub Desktop.
Save jgusta/b683e8224e445b54ab21ead7081a088d to your computer and use it in GitHub Desktop.
Download all Twitch VOD Comments and the Avatars - Aug 30, 2021
function twitchCom
# requires jq, fish shell, wget and curl
set videoId "$argv[1]"
set clientId kimne78kx3ncx6brgo4mv6wki5h1ko
set deviceId p7vCimC0LlUFnAlNYCQVMuztj9DhSZa5
set jsonPart 0
set dirPrefix "$videoId-comments"
set partFormatted (string pad -c 0 -w 3 "$jsonPart")
set avatars "$dirPrefix/avatars"
set raw "$dirPrefix/raw"
mkdir -p "$dirPrefix" "$raw" "$avatars"
set initialJson (curl "https://api.twitch.tv/v5/videos/$videoId/comments?content_offset_seconds=0" \
-H 'authority: api.twitch.tv' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'accept-language: en-us' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H 'content-type: application/json; charset=UTF-8' \
-H 'accept: application/vnd.twitchtv.v5+json; charset=UTF-8' \
-H 'x-requested-with: XMLHttpRequest' \
-H "client-id: $clientId" \
-H "x-device-id: $deviceId" \
-H 'origin: https://www.twitch.tv' \
-H 'sec-fetch-site: same-site' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-dest: empty' \
-H 'referer: https://www.twitch.tv/')
echo $initialJson >"$raw/$videoId.part-$partFormatted.json"
set logos (echo $initialJson | jq '. | [.comments[].commenter | {display_name,logo}] | unique_by(.display_name) | .')
set nextToken (cat "$raw/$videoId.part-$partFormatted.json" | jq "._next" | string trim -c '"')
while test null != "$nextToken"
set jsonPart (math $jsonPart + 1)
set partFormatted (string pad -c 0 -w 3 "$jsonPart")
set nextJson (curl "https://api.twitch.tv/v5/videos/$videoId/comments?cursor=$nextToken" \
-H 'authority: api.twitch.tv' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'accept-language: en-us' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'content-type: application/json; charset=UTF-8' \
-H 'accept: application/vnd.twitchtv.v5+json; charset=UTF-8' \
-H 'x-requested-with: XMLHttpRequest' \
-H "client-id: $clientId" \
-H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H "x-device-id: $deviceId" \
-H 'origin: https://www.twitch.tv' \
-H 'sec-fetch-site: same-site' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-dest: empty' \
-H 'referer: https://www.twitch.tv/')
echo "$nextJson" >"$raw/$videoId.part-$partFormatted.json"
set logos (echo "$nextJson" | jq --argjson lo "$logos" '. | [.comments[].commenter | {display_name,logo}] | unique_by(.display_name) | . + $lo')
set nextToken (echo "$nextJson" | jq "._next" | string trim -c '"')
sleep 1 &
wait $last_pid
end
echo "$logos" | jq "unique_by(.display_name) | ." >"$raw/logos.json"
pushd "$avatars"
cat "../raw/logos.json" | jq -r '.[] | (.logo,.display_name)' | xargs -n2 /bin/sh -c 'wget -q -O"$2.png" "$1"' /bin/sh
popd
echo done
end
@jgusta
Copy link
Author

jgusta commented Aug 30, 2021

This is a fish shell script. You also need jq.
Stick in your own device id and client id. You can grab them by looking at your browser network requests when logged into twitch.

Usage:
fish twitchCom.fish <twitch vod id>
Or drop in your ~/.config/fish/functions folder then:
twitchCom <twitch vod id>

It will create a folder in the format of <id>-comments from where you run it, then grab the raw jsons and then it will download the avatars in a sub folder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment