Skip to content

Instantly share code, notes, and snippets.

@kitz99
Created June 29, 2018 06:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kitz99/06cd208749896f368a7dbaceb3c6e781 to your computer and use it in GitHub Desktop.
Save kitz99/06cd208749896f368a7dbaceb3c6e781 to your computer and use it in GitHub Desktop.
require 'nokogiri'
require 'open-uri'
require 'json'
URL = "https://kernel-panic.me/scraping.html"
def main()
document = Nokogiri::HTML(open(URL))
results = []
products = document.xpath("//ul[@class='products']/li")
products.each do |p|
json_product = {}
json_product[:title] = p.xpath("div/div[@class='product-header']").text.strip
json_product[:description] = p.xpath("div/div[@class='product-body']/p[contains(@class, 'description')]").text.strip
json_product[:price] = p.xpath("div/div[@class='product-footer']/p[contains(@class, 'price')]").text.strip
results << json_product
end
puts results.to_json
end
main()
# output:
[
{
"description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed ut quam lectus. Sed ultrices, est nec varius vestibulum, orci\n mi placerat nisi, et accumsan dui nibh tincidunt ex.",
"price": "50.50 EUR",
"title": "Product title 1"
},
{
"description": "Proin efficitur libero mi, ut facilisis orci faucibus vel. Mauris sit amet velit semper, laoreet neque a, laoreet nibh.",
"price": "38.50 EUR",
"title": "Product title 2"
},
{
"description": "Proin efficitur libero mi, ut facilisis orci faucibus vel. Mauris sit amet velit semper, laoreet neque a, laoreet\n nibh.",
"price": "66.25 EUR",
"title": "Product title 3"
},
{
"description": "Proin efficitur libero mi, ut facilisis orci faucibus vel. Mauris sit amet velit semper, laoreet neque a, laoreet\n nibh.",
"price": "18.25 EUR",
"title": "Product title 3"
},
{
"description": "Proin efficitur libero mi, ut facilisis orci faucibus vel. Mauris sit amet velit semper, laoreet neque a, laoreet\n nibh.",
"price": "21.66 EUR",
"title": "Product title 3"
},
{
"description": "Proin efficitur libero mi, ut facilisis orci faucibus vel. Mauris sit amet velit semper, laoreet neque a, laoreet\n nibh.",
"price": "19.75 EUR",
"title": "Product title 3"
},
{
"description": "Proin efficitur libero mi, ut facilisis orci faucibus vel. Mauris sit amet velit semper, laoreet neque a, laoreet\n nibh.",
"price": "11.5 EUR",
"title": "Product title 3"
}
]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment