Skip to content

Instantly share code, notes, and snippets.

@mislav

mislav/Gemfile

Forked from janko/benchmark.rb
Last active Aug 29, 2015
Embed
What would you like to do?

If you want to properly benchmark speed of Ruby scripts, then you have to compensate for factors that affect execution time:

  1. Potential overhead of Ruby version manager
  2. Startup time of ruby interpreter
  3. System cache that can affect Ruby interpreter execution on first run.
  4. Potential extra requires in RUBYOPT environment variable.
  5. RubyGems load time
  6. Load time for individual gems at require time

For instance, a simple Ruby require that loads a gem can vary wildly in speed depending on how many system gems the user has installed on their computer.

To compensate for the above factors:

  1. Use the absolute path to the ruby executable.
  2. Measure ruby interpreter overhead and substract it from other measurement.
  3. Prime the system cache by exercising the script prior to measurement.
  4. Reset RUBYOPT and RUBYLIB to known values.
  5. It's best to avoid RubyGems altogether with --disable-gems.
  6. Because RubyGems is disabled, simply add the few libraries you need (fetched by Bundler) to RUBYLIB to solve requires.

The bench script here implements all of these solutions.

Results

On my MBP, Ruby 2.1.5, the performance of Minitest vs. RSpec is:

measuring minitest.rb
mean:    0.084
stdev:   0.003
measuring rspec.rb
mean:    0.081
stdev:   0.004

RSpec is rougly 3ms faster— a completely negligible difference.

An absolute number is much more informative than percentages here. It doesn't make sense to compare percentages when you're measuring speed gain if you don't know the absolute number.

But it doesn't matter.

This benchmark is a poor comparison of performance of these testing frameworks because it's essentially a Hello World program. These test scripts aren't representative of the actual way how these testing frameworks would get used. Usually in tests, the startup time of a library isn't so much of a problem, but its speed in executing various assertions and handling huge test cases is much more relevant.

However, in the end, it will be your own slow code that will most likely be slowing down the test suite. Not the test library.

#!/usr/bin/env bash
set -e
count=50
unset RUBYPATH
export RUBYOPT="--disable-gems"
export TIMEFORMAT='%3R'
if [ ! -d ./bundle ]; then
bundle install --standalone
fi
RUBYLIB=""
for lib in bundle/ruby/*/gems/*/lib; do
RUBYLIB="${RUBYLIB}:${lib}"
done
export RUBYLIB
stdev() {
ruby -e '
values = []
ARGF.each { |line| values << line.to_f if line =~ /\d/ }
sum = values.inject(:+)
mean = sum.to_f / values.size
diffs = values.map {|v| (v - mean) ** 2 }
stdev = Math.sqrt(diffs.inject(:+).to_f / values.size)
puts "mean: %7.3f" % mean
puts "stdev: %7.3f" % stdev
' "$@"
}
if type -p rbenv >/dev/null; then
ruby="$(rbenv which ruby)"
else
ruby="$(which ruby)"
fi
measure() {
local n="$count"
# measure the overhead of executing ruby directly
local baseline=$( (time "$ruby" -e0) 2>&1 )
# prime system cache and ensure that the script actually works
"$ruby" "$@" >/dev/null
echo "measuring" "$@"
{ while [ "$n" -gt 0 ]; do
echo $( (time "$ruby" "$@" >/dev/null) 2>&1 ) - $baseline | bc
n=$((n - 1))
done
} | stdev
}
measure "minitest.rb"
measure "rspec.rb"
source "https://rubygems.org"
gem "minitest"
gem "rspec"
GEM
remote: https://rubygems.org/
specs:
diff-lcs (1.2.5)
minitest (5.5.1)
rspec (3.2.0)
rspec-core (~> 3.2.0)
rspec-expectations (~> 3.2.0)
rspec-mocks (~> 3.2.0)
rspec-core (3.2.3)
rspec-support (~> 3.2.0)
rspec-expectations (3.2.1)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.2.0)
rspec-mocks (3.2.1)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.2.0)
rspec-support (3.2.2)
PLATFORMS
ruby
DEPENDENCIES
minitest
rspec
require "minitest/autorun"
class MintestTest < Minitest::Test
def test_foo
assert true
end
end
require "rspec/autorun"
RSpec.describe "RSpec" do
it "foo" do
expect(true).to eq(true)
end
end
@abinoam

This comment has been minimized.

Copy link

@abinoam abinoam commented Apr 10, 2015

MacBook pro 2,4GHz i5, 16Gb.

$ ./bench
measuring minitest.rb
mean:    0.097
stdev:   0.006
measuring rspec.rb
mean:    0.123
stdev:   0.006
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.