Parallel Requests w/ Faraday + Typhoeus

faraday_typh_default.rb

require "faraday"
require 'typhoeus'

conn = Faraday.new(:url => 'http://httpstat.us') do |builder|
  builder.request  :url_encoded
  builder.response :logger
  builder.adapter  :typhoeus
end

conn.in_parallel do
 # will use Hydra's default settings: max_concurrency: 200, memoization, initial request pool size: 10
end

faraday_typh_settings.rb

require "faraday"
require 'typhoeus'

HYDRA = Typhoeus::Hydra.new(:max_concurrency => 5)
HYDRA.disable_memoization

conn = Faraday.new(:url => "http://httpstat.us", :parallel_manager => HYDRA) do |builder|
  builder.request  :url_encoded
  builder.adapter  :typhoeus
end

conn.in_parallel do
  # Will use user-defined Hydra settings: max_concurrency: 5, no memoization
end

Faraday Parallel Request Docs: https://github.com/technoweenie/faraday/wiki/Parallel-requests

The first example will use Typhoeus::Hydra w/ default settings:

• max concurrency of 200
• starting request pool size of 10
• request memoization

The second example shows how to override Typhoeus::Hydra defaults w/ a custom parallel_manager

See Typhoeus::Hydra

great! Thanks ;)

New location for faraday Parallel requests wiki: https://github.com/lostisland/faraday/wiki/Parallel-requests