Skip to content

Instantly share code, notes, and snippets.

@jjaaccoobb
Last active February 2, 2020 22:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jjaaccoobb/9ba747f34d86c038e011bdc8fcf750cf to your computer and use it in GitHub Desktop.
Save jjaaccoobb/9ba747f34d86c038e011bdc8fcf750cf to your computer and use it in GitHub Desktop.
Node script to recursively fetch a user's comments until done or API quota exceeded
const fetch = require('node-fetch');
const cheerio = require('cheerio');
(async () => {
let nextPage = 'https://old.reddit.com/user/fbiciansa.gov';
let pageNo = 1;
while (nextPage) {
const response = await fetch(nextPage);
const body = await response.text();
const $ = cheerio.load(body);
const $comments = $('div.thing');
console.log(`Fetching page #${pageNo}\n`);
const texts = $comments.map((idx, el) => {
const commentText = $(el)
.find('.entry .usertext-body')
.text()
.trim();
const timestamp = $(el)
.find('.entry .live-timestamp')
.attr('datetime');
const likes =
$(el)
.find('.entry .score.dislikes')
.attr('title') || 0;
const dislikes =
$(el)
.find('.entry .score.likes')
.attr('title') || 0;
const unvoted =
$(el)
.find('.entry .score.unvoted')
.attr('title') || 0;
const permalink = $(el).data('permalink');
console.log(commentText, timestamp, likes, dislikes, unvoted, permalink, '\n');
});
nextPage = $('.next-button a').attr('href') || false;
pageNo++;
}
})();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment