Skip to content

Instantly share code, notes, and snippets.

@jlstr
Created October 24, 2012 20:13
Show Gist options
  • Save jlstr/3948548 to your computer and use it in GitHub Desktop.
Save jlstr/3948548 to your computer and use it in GitHub Desktop.
Parallel
def crawl_in_parallel(urls=[], type, subcategory)
hydra = Typhoeus::Hydra.new(:max_concurrency => 5)
urls.each do |url|
request = Typhoeus::Request.new(url,
timeout: 7000,
follow_location: true,
)
request.on_complete do |response|
if response.success?
begin
ShopbopProductWrapper.new(url, type, subcategory, response.body)
rescue Exception => e
print "F (#{e.message})"
end
elsif response.timed_out?
puts "TIMEOUT for: #{url}"
elsif response.code == 0
puts "FAILED HTTP RESPONSE FOR: #{url}"
else
puts "HTTP request failed: #{response.code.to_s}"
end
end
hydra.queue request
end
hydra.run
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment