Skip to content

Instantly share code, notes, and snippets.

@ngauthier
Created March 12, 2012 01:14
Show Gist options
  • Save ngauthier/2019132 to your computer and use it in GitHub Desktop.
Save ngauthier/2019132 to your computer and use it in GitHub Desktop.
Ruby Workflow Benchmark

Ruby Workflow System Benchmark

Why?

I've chatted with people that love working on their macbook airs, and honestly, they have a lot going for them. They're nice light machines with a surprising amount of performance for their form factor.

I always argue that they're way slower than much cheaper machines, like a good workstation. Ergonomics aside, I wanted to get some performance numbers.

I've attached a script to this gist that will grab Sinatra and time how long it takes to bundle it, run the full tests, and run a single test.

This is a pretty lightweight project, so times should be short all-around. I'd like to do this with a rails app as well, but couldn't quickly find a medium size open source rails app.

OK, what should I do?

Please run the script, and in the comments fill out the following:

  • Computer type (Macbook Air, custom workstation, Thinkpad, etc)
  • Operating System
  • Hard disk (or SSD) type (and throughput if you know it)
  • Processor type and speed
  • Amount of RAM
  • Anything else you think contributes to the performance.

Get your results

Then, include the time line for each command. It should look something like:

0.56user 0.07system 0:00.64elapsed 98%CPU (0avgtext+0avgdata 103648maxresident)k

The first number is the most important here.

Compute your score

Now, for a completely arbitrary score, do the following computation:

(Bundle Time) + (Suite Time * 10) + (Test Time * 100)

To reflect a rough workflow of bundling once, running the suite before commits, and running individual files while developing.

Final Note

This benchmark is probably pretty flawed. Before complaining about potential bias or problems, come up with your own benchmark and let's try it.

#!/bin/sh
# Please use ruby-1.9.3-p0-falcon for best accuracy
git clone https://github.com/sinatra/sinatra.git
cd sinatra
git checkout 1.3.2
time bundle --path .bundle/gems
time bundle exec rake
time bundle exec ruby test/base_test.rb
@ngauthier
Copy link
Author

OK, here's my results:

  • 4 year old custom workstation. About $600 (sans gaming video card :-) )
  • Ubuntu 11.10 x64
  • 1TB platter hard disk
  • AMD Phenom(tm) II X4 940 Processor @ 3.00ghz
  • 8GB RAM
15.11user 2.34system 0:32.56elapsed 53%CPU (0avgtext+0avgdata 304192maxresident)k
4.11user 0.23system 0:05.22elapsed 83%CPU (0avgtext+0avgdata 328944maxresident)k
0.56user 0.07system 0:00.64elapsed 98%CPU (0avgtext+0avgdata 103648maxresident)k

Score: 148.76

@llimllib
Copy link

Did all the Sinatra tests pass for you? They didn't for me, so that probably skews the results.

Unibody 17" Macbook Pro

real    1m43.539s    user   1m14.033s sys   0m5.747s
real    2m6.654s user   0m5.915s sys    0m0.786s
real    0m0.649s user   0m0.535s sys    0m0.105s

Which time did you use for calculating the scores?

@ngauthier
Copy link
Author

ngauthier commented Mar 12, 2012 via email

@llimllib
Copy link

113 + 1260 + 649 = 2022 and it seems highly suspect to me that your computer is twice an order of magnitude faster than mine.

@ngauthier
Copy link
Author

ngauthier commented Mar 12, 2012 via email

@MattRogish
Copy link

Here are my results (note this is 1.9.3p125 since I couldn't get p0 working on OSX Lion)

  • 13" MacBook Air (Mid 2011)
  • OSX 10.7.3
  • 1.7GHz i5
  • 4GB RAM
  • 128GB SSD

real 0m23.611s user 0m15.345s sys 0m3.134s

real 0m4.989s user 0m3.844s sys 0m0.374s

real 0m0.697s user 0m0.580s sys 0m0.108s

(23.611) + (4.989 * 10) + (0.697 * 100) = 143.201 real

@nerdEd
Copy link

nerdEd commented Mar 12, 2012

First Gen MBA: 2.13 GHz Intel Core 2 Duo, 4GB RAM, SSD

real    0m35.559s
user    0m21.196s
sys     0m4.396s

real    0m6.560s
user    0m5.225s
sys     0m0.493s

real    0m0.891s
user    0m0.762s
sys     0m0.116s

(35.559) + (6.560 * 10) + (0.891 * 100) = 190.259

@MattRogish
Copy link

Nick,

How did you get 112? Looks like you were using user and not elapsed (OSX's version is real)? If I use user, my score drops to 111.78

@f3nry
Copy link

f3nry commented Mar 12, 2012

Here's my results:

  • Spring 2011 MacBook Pro
  • 2 GHz Intel Core i7
  • Mac OSX 10.7.3
  • 8 GB 1333 MHz DDR3
  • 500 GB HD
  • AMD Radeon HD 6490M 256 MB

Bundler

real    1m3.902s
user    0m24.400s
sys 0m3.082s

Rake:

Finished tests in 2.524731s, 318.0537 tests/s, 618.2837 assertions/s.

803 tests, 1561 assertions, 0 failures, 0 errors, 0 skips

real    0m4.736s
user    0m3.677s
sys 0m0.306s

ruby test/base_test.rb

Finished tests in 0.010234s, 1367.9891 tests/s, 2540.5511 assertions/s.

14 tests, 26 assertions, 0 failures, 0 errors, 0 skips

real    0m0.737s
user    0m0.649s
sys 0m0.079s

Score: 184.96 (by seconds)

@nerdEd
Copy link

nerdEd commented Mar 12, 2012

Might also be worth making sure that everyone is using the latest version of bundler, or better yet just ignoring the bundler time.

@f3nry
Copy link

f3nry commented Mar 12, 2012

Bundler is also somewhat dependent on network speed, so that may not be a relevant test to computer speed.

@ngauthier
Copy link
Author

Dell XPS 13

Intel(R) Core(TM) i5-2467M CPU @ 1.60GHz, SSD, Ubuntu 4GB RAM.

17.64user 2.43system 0:37.21elapsed 53%CPU (0avgtext+0avgdata 269856maxresident)k
4.78user 0.36system 0:05.83elapsed 88%CPU (0avgtext+0avgdata 322064maxresident)k
0.80user 0.10system 0:00.92elapsed 98%CPU (0avgtext+0avgdata 108672maxresident)k

Score 185.51

@ngauthier
Copy link
Author

@MattRogish oops yeah. Sorry time changes formats when run as a script and I got confused. Updated.

@ngauthier
Copy link
Author

Yeah good call on bundler.

I'm going to try to setup a better benchmark with a medium size rails project if I can find one. This is preliminary :-)

@abachman
Copy link

real    0m26.267s
user    0m12.111s
sys   0m3.117s

real    0m4.661s
user    0m3.542s
sys   0m0.449s

real    0m0.450s
user    0m0.356s
sys   0m0.086s

Total score: 26.27 + (4.66 * 10) + (0.45 * 100) =

117.87

April 2011 15" MacBook Pro

  • APPLE SSD TS128C
  • 8GB 1333MHz DDR3
  • 2 GHz Intel Core i7

Enjoy configuring your video drivers, Linux workstation guy.

;|

@ngauthier
Copy link
Author

I love configuring video cards

@jszmajda
Copy link

Netbook I use for fun:

  • Lenovo S12
  • Ubuntu 11.10 x32
  • Atom N270 @ 1.60GHz
  • Fujitsu MHZ2160BH 160Gb 5200 rpm HD
  • 2Gb RAM

bundle 72.50user 8.01system 1:29.32elapsed 90%CPU (0avgtext+0avgdata 210288maxresident)k
suite 18.30user 0.91system 0:21.44elapsed 89%CPU (0avgtext+0avgdata 205568maxresident)k
test 2.64user 0.25system 0:02.92elapsed 99%CPU (0avgtext+0avgdata 80896maxresident)k
#595.72 (lulz)

My machine at the office:

  • 1 year old custom-ish cyberpower desktop
  • Ubuntu 11.10 x64
  • Core 2 Duo i7 @ 3.07 Ghz
  • Kingston SNV425S264GB SSD (200 mbps read, 110 mbps write)
  • 12Gb RAM·

bundle 13.16user 2.00system 0:19.28elapsed 78%CPU (0avgtext+0avgdata 281280maxresident)k
suite 3.02user 0.17system 0:03.54elapsed 90%CPU (0avgtext+0avgdata 332832maxresident)k
test 0.43user 0.04system 0:00.49elapsed 97%CPU (0avgtext+0avgdata 103680maxresident)k
#103.68

Does that mean I win? ;)

@bryanl
Copy link

bryanl commented Mar 12, 2012

On my i5 Macbook Air: 27.75 + (5.66_10) + (0.90_100) = 174.35
On my i7 2600 desktop (intel ssd + 8GB of memory): 13.6 + (2.64_10) + (0.38_100) = 78

Not sure what this proves. My desktop is way faster than my macbook air.

@ngauthier
Copy link
Author

ngauthier commented Mar 12, 2012 via email

@ngauthier
Copy link
Author

My 2 year old $400 workstation. This is my "work" computer since it has a higher processor and ssd (and it lives at work).

  • Ubuntu 11.10
  • AMD Phenom(tm) II X4 965 Processor @ 3.4ghz
  • 8GB RAM
  • SSD

16.06user 2.90system 0:25.09elapsed 75%CPU (0avgtext+0avgdata 269536maxresident)k
3.55user 0.20system 0:04.23elapsed 88%CPU (0avgtext+0avgdata 304736maxresident)k
0.53user 0.04system 0:00.57elapsed 99%CPU (0avgtext+0avgdata 103664maxresident)k

Score: 124.39

@MattRogish
Copy link

My work computer:

  • 15" MacBook Pro - Mid 2010
  • 2.4GHz i5
  • 4GB RAM
  • 256GB SSD

I got creamed in the install which I hope was network related (work network is inconsistent).

real 1m38.231s user 0m27.003s sys 0m4.128s

real 0m5.291s user 0m4.045s sys 0m0.499s

real 0m0.875s user 0m0.722s sys 0m0.140s
#98.231 + (5.291 * 10) + (0.875 * 100) = 238.641

@ngauthier
Copy link
Author

My new ivy bridge machine, overclocked!

  • Ubuntu 12.04
  • Intel Ivy Bridge i7-3770k OCed to 3.8ghz (4.2ghz turbo)
  • 16GB RAM
  • Intel SATA III 6gb/s SSD

6.90user 1.10system 0:12.35elapsed 64%CPU (0avgtext+0avgdata 267648maxresident)k
2.51user 0.17system 0:03.08elapsed 87%CPU (0avgtext+0avgdata 259088maxresident)k
0.32user 0.02system 0:00.35elapsed 96%CPU (0avgtext+0avgdata 109536maxresident)k
#78.15 :-D

@kmarsh
Copy link

kmarsh commented Jun 16, 2012

Fancy new base-level MacBook Pro with Retina Display. Mac OS X 10.7.4, ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.4.2], 2.3GHz quad-core Intel Core i7 (Ivy Bridge), 8GB RAM, 256GB SSD:

15.385 + (4.329_10) + (.526_100) = 111.275

Honestly not as impressive as I'd thought... the text in the terminal was nice and sharp, though! ;)

@MattRogish
Copy link

Here's my brand-new 2012 MBA w/2GHz Ivy Bridge, 8GB RAM, 256GB SSD:

real 0m13.498s
user 0m8.516s
sys 0m2.009s

real 0m4.146s
user 0m3.314s
sys 0m0.378s

real 0m0.591s
user 0m0.488s
sys 0m0.095s

13.498 + (4.146_10) + (0.591_100) = 114.83800

The bundle install numbers are likely skewed by network stuff - but clearly your test suite runs a lot faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment