Skip to content

Instantly share code, notes, and snippets.

@Suleman-Elahi
Created March 27, 2019 09:21
Show Gist options
  • Save Suleman-Elahi/73727b85b286bccabe6b79b821ce59f2 to your computer and use it in GitHub Desktop.
Save Suleman-Elahi/73727b85b286bccabe6b79b821ce59f2 to your computer and use it in GitHub Desktop.
Scrape a List of Topics from a Subreddit Using Bash (from HowToGeek)
#!/bin/bash
if [ -z "$1" ]
then
echo "Please specify a subreddit"
exit 1
fi
SUBREDDIT=$1
NOW=$(date +"%m_%d_%y-%H_%M")
OUTPUT_FILE="${SUBREDDIT}_${NOW}.txt"
curl -s -A "bash-scrape-topics" https://www.reddit.com/r/${SUBREDDIT}.json | \
jq '.data.children | .[] | .data.title, .data.url, .data.permalink' | \
while read -r TITLE; do
read -r URL
read -r PERMALINK
echo -e "${TITLE}\t${URL}\t${PERMALINK}" | tr --delete \" >> ${OUTPUT_FILE}
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment