Skip to content

Instantly share code, notes, and snippets.

@fcheung
Created June 12, 2012 21:08
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save fcheung/92d77b62eb570031823d to your computer and use it in GitHub Desktop.
Save fcheung/92d77b62eb570031823d to your computer and use it in GitHub Desktop.
tire multisearch implementation
module Tire
module Search
class Search
attr_reader :types
end
class MultiSearch
def initialize(searches, options ={})
@searches = searches
@options = options
end
def to_payload
payload = @searches.each_with_object([]) do |search, result|
result << {:index => search.indices, :types => search.types}.to_json
result << search.to_json
end
payload << ""
payload.join("\n")
end
def results
@results || (perform; @results)
end
def perform
response = Configuration.client.post("#{Configuration.url}/_msearch", to_payload)
json = MultiJson.decode(response.body)
@results = json['responses'].each_with_index.collect do |response_json, index|
Results::Collection.new(response_json, @options.merge(@searches[index].options))
end
end
end
end
end
@karmi
Copy link

karmi commented Jun 13, 2012

I'm not sure that a wrapper would add a lot here (...)

True. It could add some consistency for iteration; but yes, each is already there for Hash and Array...

@romanbsd
Copy link

I think that it would be nice to have both the declarative form (my version), as well as assembled ad-hoc form (as in this gist). I'd also add the << method to the msearch above.

@karmi
Copy link

karmi commented Jun 13, 2012

@fcheung: By the way, is there any observable/measurable benefit from multi search, compared to HTTP client which uses persistent connection?

@fcheung
Copy link
Author

fcheung commented Jun 13, 2012

I don't know to be honest - haven't tried that yet

@karmi
Copy link

karmi commented Jun 13, 2012

@fcheung: Would be awesome if you could do a simple benchmark à la https://gist.github.com/1204159 ...

@fcheung
Copy link
Author

fcheung commented Jun 13, 2012

Can do - didn't realise that tire already had support for multiple http clients built-in

@fcheung
Copy link
Author

fcheung commented Jun 13, 2012

I'll make a proper reproducible benchmark later on (I'll also look into using ipfw to simulate the difference as latency increases), but just plugging this into my app I get around 120ms using rest client, individual searches, around 80ms using curb, 30ms using rest client + msearch, and curb + msearch is around the same.

@romanbsd
Copy link

@fcheung it was just recently merged :) We're using it with net-http-persistent, and meanwhile quite happy with it. We verified that it indeed uses persistent connections.

@karmi
Copy link

karmi commented Jun 13, 2012

it was just recently merged

@romanbsd: No, Tire had support for RestClient and Curb for a long long time (karmi/retire@8010531)

@karmi
Copy link

karmi commented Jun 13, 2012

@fcheung: Those numbers sound very interesting.

@romanbsd
Copy link

@karmi oops, I was referring to persistent connections, I didn't succeed in achieving persistent conns before that.

@karmi
Copy link

karmi commented Jun 13, 2012

@romanbsd: Curb definitely does persistence connections, see the benchmark...

@romanbsd
Copy link

I was checking it with tcpdump at the time, and was observing SYN packets. Perhaps there was something different in my setup... Is curb's persistent connection handling thread safe?

@agnellvj
Copy link

Will this get rolled into Tire? For now I am gonna use the bits I need to make a multisearch request via RestClient manually until that happens.

@karmi
Copy link

karmi commented Nov 11, 2012

Yes, multi search support is now available in Tire 0.5.0/master:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment