Skip to content

Instantly share code, notes, and snippets.

@arabyniuk
Created May 2, 2016 12:49
Show Gist options
  • Save arabyniuk/ae4848f467ccea9580f8bd829ac35723 to your computer and use it in GitHub Desktop.
Save arabyniuk/ae4848f467ccea9580f8bd829ac35723 to your computer and use it in GitHub Desktop.
class Scraper
def self.scrape_reviewer_profile_url
User.find_each do |user|
next unless user.amazon_profile_url || !user.amazon_profile_url.blank?
begin
browser = Watir::Browser.new :firefox
browser.goto user.amazon_profile_url
sleep 2
browser.span(class: 'a-expander-prompt').click
data = Nokogiri::HTML(browser.html)
result = data.css('a.top-reviewer-link div.a-row span.a-size-base')
.text.sub('#', '')
user.update_attribute(:amazon_reviewer_rank, result)
browser.close
rescue Exception => e
puts "--------------"
puts e
puts "could not scrap page for user ##{user.id}"
puts "--------------"
browser.close if browser
next
end
end
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment