Skip to content

Instantly share code, notes, and snippets.

Created January 7, 2019 05:47
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do? top posts scraping script
// navigate to and scroll untill postsData's length is more than 500
postsCount = 500
postsData = [...document.querySelectorAll('#substories .single-article')]
.slice(0, postsCount)
.map(post => {
try {
const [author, date] = post.querySelector('h4').textContent.trim().split('・')
const postData = {
title: post.querySelector('h3').textContent.trim(),
tags: post.querySelector('.tags').textContent.trim().split(/\s+/),
reactionsCount: parseInt(post.querySelector('.reactions-count .engagement-count-number').textContent.trim(), 10),
commentsCount: parseInt(post.querySelector('.comments-count .engagement-count-number').textContent.trim(), 10),
return postData;
} catch (e) {
// There are two ill formatted posts and we are going to ignore them
JSON.stringify(postsData, null, 2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment