Skip to content

Instantly share code, notes, and snippets.

@havenwood
Created August 17, 2022 01:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save havenwood/996d4eca580526202d0e401dfa82aded to your computer and use it in GitHub Desktop.
Save havenwood/996d4eca580526202d0e401dfa82aded to your computer and use it in GitHub Desktop.
require 'nokogiri'
require 'open-uri'
require 'pry'
module Scraper
module_function
def fetch_frontpage(frontpage)
raw = URI.read(frontpage)
parsed = Nokogiri::HTML(raw)
target = parsed.css('a.titlelink')
links = target.map do |link|
link.attribute('href').value
end
create_posts(links)
end
def create_posts(links)
links.map do |post|
puts post
raw = URI.read(post)
parsed = Nokogiri::HTML(raw)
p parsed
end
end
end
Scraper.fetch_frontpage('https://news.ycombinator.com')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment