Testing throughput bottlenecks in Goliath from Mongoid gem

Pre-requisites

0. gem install goliath (should be v0.9.4 as of writing this)
1. gem install mongoid (should be v2.3.3 as of writing this)
2. create and save test.rb file (see below)
3. % ruby test.rb -sv
4. % ab -n500 -c10 http://localhost:9000/status
5. note the req/s results (somewhere around 80-90 req/s on my machine)
6. edit the test.rb file, remove the 'require mongoid' line
7. repeat steps 3 and 4.
8. note the req/s results (somewhere aaround 550 req/s on my machine)


Conclusion: Mongoid is patching something in its initialization that is causing severe performance bottlenecks in Goliath.

test.rb

$:<< '../lib' << 'lib'

require 'goliath'
require 'mongoid'

class HelloWorld < Goliath::API
  use Goliath::Rack::Heartbeat

  def response(env)
    [200, {}, ""]
  end
end

ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-darwin10.7.0]

ruby test.rb -e production
ab -c 10 -n500 http://localhost:9000/

Concurrency Level:      10
Time taken for tests:   0.317 seconds
Complete requests:      500
Failed requests:        0
Write errors:           0
Total transferred:      46552 bytes
HTML transferred:       0 bytes
Requests per second:    1575.94 [#/sec] (mean)
Time per request:       6.345 [ms] (mean)
Time per request:       0.635 [ms] (mean, across all concurrent requests)
Transfer rate:          143.29 [Kbytes/sec] received

With / without mongoid it's about the same. Hmm?

Adam, what ruby version are you using?

Sujal, I tried this under both of these versions:

ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-darwin11.2.0]
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.1.0]

The performance difference is due to running goliath without the "-e production" flag. Problem solved! (Thanks, Ilya!)

Have you done any testing when a query is involved? I'm not getting anywhere near this kind of performance when talking to the DB.

Hi Sujal, Yes, I have tested with actual queries in a Geo-spatially-indexed collection. The performance isn't nearly that of a simple status/ping call... but I am able to achieve around 500-600 rps on my local machine (with the mongo db also running locally).

…

On Wed, Nov 16, 2011 at 9:12 AM, sujal < ***@***.*** > wrote: Have you done any testing when a query is involved? I'm not getting anywhere near this kind of performance when talking to the DB. --- Reply to this email directly or view it on GitHub: https://gist.github.com/1367854

Thanks for answering all my questions. I really appreciate it.

The thing I'm puzzling through is how real that 500-600 number you're citing is. The key is that your test isn't really taking advantage of what async frameworks bring to the table. Since all your data is running on a local box, IO latency is negligible, so I would expect your test to perform as fast as a threaded server. In other words, if your requests take 2ms, you should get 500rps. If they take less, you should get even more.

I've been testing with MongoDB running across a WAN (from my box here at my friends house to my server at home) and there performance takes a hit. I'm still trying to validate my testing (I just sent a message to the goliath list), so take this with a grain of salt. For example, my performance issues last night turned out to be a bug in apache bench on OS X Lion. ;-)

Sujal

(the short version of what I'm puzzling through is how do I know that MongoDB calls are really async... that's where this all started)