README.md

If you want to properly benchmark speed of Ruby scripts, then you have to compensate for factors that affect execution time:

1. Potential overhead of Ruby version manager
2. Startup time of ruby interpreter
3. System cache that can affect Ruby interpreter execution on first run.
4. Potential extra requires in RUBYOPT environment variable.
5. RubyGems load time
6. Load time for individual gems at require time

For instance, a simple Ruby require that loads a gem can vary wildly in speed depending on how many system gems the user has installed on their computer.

To compensate for the above factors:

1. Use the absolute path to the ruby executable.
2. Measure ruby interpreter overhead and substract it from other measurement.
3. Prime the system cache by exercising the script prior to measurement.
4. Reset RUBYOPT and RUBYLIB to known values.
5. It's best to avoid RubyGems altogether with --disable-gems.
6. Because RubyGems is disabled, simply add the few libraries you need (fetched by Bundler) to RUBYLIB to solve requires.

The bench script here implements all of these solutions.

Results

On my MBP, Ruby 2.1.5, the performance of Minitest vs. RSpec is:

measuring minitest.rb
mean:    0.084
stdev:   0.003
measuring rspec.rb
mean:    0.081
stdev:   0.004

RSpec is rougly 3ms faster— a completely negligible difference.

An absolute number is much more informative than percentages here. It doesn't make sense to compare percentages when you're measuring speed gain if you don't know the absolute number.

But it doesn't matter.

This benchmark is a poor comparison of performance of these testing frameworks because it's essentially a Hello World program. These test scripts aren't representative of the actual way how these testing frameworks would get used. Usually in tests, the startup time of a library isn't so much of a problem, but its speed in executing various assertions and handling huge test cases is much more relevant.

However, in the end, it will be your own slow code that will most likely be slowing down the test suite. Not the test library.

bench

#!/usr/bin/env bash
set -e

count=50

unset RUBYPATH
export RUBYOPT="--disable-gems"
export TIMEFORMAT='%3R'

if [ ! -d ./bundle ]; then
  bundle install --standalone
fi

RUBYLIB=""
for lib in bundle/ruby/*/gems/*/lib; do
  RUBYLIB="${RUBYLIB}:${lib}"
done
export RUBYLIB

stdev() {
  ruby -e '
    values = []
    ARGF.each { |line| values << line.to_f if line =~ /\d/ }

    sum   = values.inject(:+)
    mean  = sum.to_f / values.size
    diffs = values.map {|v| (v - mean) ** 2 }
    stdev = Math.sqrt(diffs.inject(:+).to_f / values.size)

    puts "mean:  %7.3f" % mean
    puts "stdev: %7.3f" % stdev
  ' "$@"
}

if type -p rbenv >/dev/null; then
  ruby="$(rbenv which ruby)"
else
  ruby="$(which ruby)"
fi

measure() {
  local n="$count"

  # measure the overhead of executing ruby directly
  local baseline=$( (time "$ruby" -e0) 2>&1 )

  # prime system cache and ensure that the script actually works
  "$ruby" "$@" >/dev/null

  echo "measuring" "$@"
  { while [ "$n" -gt 0 ]; do
      echo $( (time "$ruby" "$@" >/dev/null) 2>&1 ) - $baseline | bc
      n=$((n - 1))
    done
  } | stdev
}

measure "minitest.rb"
measure "rspec.rb"

Gemfile

source "https://rubygems.org"

gem "minitest"
gem "rspec"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.2.5)
    minitest (5.5.1)
    rspec (3.2.0)
      rspec-core (~> 3.2.0)
      rspec-expectations (~> 3.2.0)
      rspec-mocks (~> 3.2.0)
    rspec-core (3.2.3)
      rspec-support (~> 3.2.0)
    rspec-expectations (3.2.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.2.0)
    rspec-mocks (3.2.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.2.0)
    rspec-support (3.2.2)

PLATFORMS
  ruby

DEPENDENCIES
  minitest
  rspec

minitest.rb

require "minitest/autorun"

class MintestTest < Minitest::Test
  def test_foo
    assert true
  end
end

rspec.rb

require "rspec/autorun"

RSpec.describe "RSpec" do
  it "foo" do
    expect(true).to eq(true)
  end
end

MacBook pro 2,4GHz i5, 16Gb.

$ ./bench
measuring minitest.rb
mean:    0.097
stdev:   0.006
measuring rspec.rb
mean:    0.123
stdev:   0.006