Skip to content

Instantly share code, notes, and snippets.

@veloper
Created November 29, 2011 17:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save veloper/1405631 to your computer and use it in GitHub Desktop.
Save veloper/1405631 to your computer and use it in GitHub Desktop.
class Uri < ActiveRecord::Base
belongs_to :job
has_many :scrapes
def get_request_object options
require 'rubygems'
require 'typhoeus'
request = Typhoeus::Request.new(self.uri, options)
# Callback Handler
request.on_complete do |response|
# Create new scrape record
scrape = self.scrapes.new
# Global Data
scrape.code = response.code # http status code
scrape.time = response.time # time in seconds the request took
# response.headers # the http headers
# response.headers_hash # http headers put into a hash
# Check response
if response.success?
# Save content
scrape.content = response.body
elsif response.timed_out?
scrape.is_timeout = 1
elsif response.code == 0
scrape.error = "Could not get an http response, something's wrong: " + response.curl_error_message
else
scrape.error = "HTTP request failed, see code."
end
# Save the scrape record.
scrape.save
end
return request
end
...
# In Daemon
hydra = Typhoeus::Hydra.new
uri_requests.each do |hash|
# Save Requested At
hash[:uri].requested_at = requested_at
hash[:uri].save
hydra.queue hash[:request]
end
hydra.run # Blocks!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment