Skip to content

Instantly share code, notes, and snippets.

@gperrin01
Created May 22, 2017 18:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gperrin01/9fb1be1b60b97ba6644c7f527cbe3111 to your computer and use it in GitHub Desktop.
Save gperrin01/9fb1be1b60b97ba6644c7f527cbe3111 to your computer and use it in GitHub Desktop.
Scrape Hacker News first page - Sample code to show async/await coding
nick.newTab().then(async (tab) => {
await tab.open("news.ycombinator.com")
await tab.untilVisible("#hnmain") // We make sure we have loaded the page
await tab.inject("../injectables/jquery-3.0.0.min.js") // We're going to use jQuery to scrape
const hackerNewsLinks = await tab.evaluate((arg, callback) => {
const data = []
$(".athing").each((index, element) => {
data.push({
title: $(element).find(".storylink").text(),
url: $(element).find(".storylink").attr("href")
})
})
callback(null, data)
})
console.log(JSON.stringify(hackerNewsLinks, null, 3))
})
.then(() => {
console.log("Job done!")
nick.exit()
})
.catch((err) => {
console.log(`Something went wrong: ${err}`)
nick.exit(1)
})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment