Benchmarking Redis drivers: Ruby gem, Hiredis, and a custom Ruby implementation

000-highlights.md

Drivers

Name	Description
ruby	The redis gem with the plain-Ruby driver
hiredis	The redis gem with the hiredis driver (compiled as a C extension)
myredis	Pure-Ruby, optimized for the specific operation

Benchmarks

Cache hit (returns a 2KB string)

Iterations per wall-clock second (higher is better):

Implementation	i/s
myredis	28,952.8
hiredis	27,134.9 (7% slower)
ruby	19,490.2 (49% slower)

CPU time to run 100k iterations (lower is better):

Implementation	CPU seconds
myredis	1.955189
hiredis	2.296374 (17% slower)
ruby	4.189748 (114% slower)

Cache miss (returns nil)

Iterations per wall-clock second (higher is better):

Implementation	i/s
myredis	30,847.7
hiredis	29,121.0 (6% slower)
ruby	21,847.2 (41% slower)

CPU time to run 100k iterations (lower is better):

Implementation	CPU seconds
myredis	1.881291
hiredis	2.143884 (14% slower)
ruby	3.486066 (85% slower)

001-raw-results.txt

Fetching gem metadata from https://rubygems.org/....
Resolving dependencies...
Using bundler 2.2.3
Using hiredis 0.6.3
Using redis 4.2.5
Using benchmark-ips 2.9.1
Cache-hit function check passed
Cache-miss function check passed

Cache hit (returns a 2KB string)
Warming up --------------------------------------
                ruby     1.496k i/100ms
             hiredis     2.767k i/100ms
             myredis     2.963k i/100ms
Calculating -------------------------------------
                ruby     19.490k (±13.3%) i/s -     95.744k in   5.051560s
             hiredis     27.135k (± 3.0%) i/s -    138.350k in   5.103274s
             myredis     28.953k (± 2.9%) i/s -    145.187k in   5.018832s

Comparison:
             myredis:    28952.8 i/s
             hiredis:    27134.9 i/s - 1.07x  (± 0.00) slower
                ruby:    19490.2 i/s - 1.49x  (± 0.00) slower


Cache hit CPU time
ruby   : 4.189748 (2.14x)
hiredis: 2.296374 (1.17x)
myredis: 1.955189 (1.00x)

Cache miss (returns nil)
Warming up --------------------------------------
                ruby     2.258k i/100ms
             hiredis     2.929k i/100ms
             myredis     3.038k i/100ms
Calculating -------------------------------------
                ruby     21.847k (±11.9%) i/s -    108.384k in   5.056322s
             hiredis     29.121k (± 3.1%) i/s -    146.450k in   5.034106s
             myredis     30.848k (± 2.4%) i/s -    154.938k in   5.025721s

Comparison:
             myredis:    30847.7 i/s
             hiredis:    29121.0 i/s - 1.06x  (± 0.00) slower
                ruby:    21847.2 i/s - 1.41x  (± 0.00) slower


Cache miss CPU time
ruby   : 3.486066 (1.85x)
hiredis: 2.143884 (1.14x)
myredis: 1.881291 (1.00x)

benchmark_redis.rb

require 'bundler/inline'

require 'socket'
class MyRedis
  CRLF = "\r\n"

  def initialize
    @connection = TCPSocket.new('localhost', 6379)
    @connection.sync = false
  end

  def get(key)
    @connection << "*2\r\n$3\r\nget\r\n$#{key.bytesize}\r\n#{key}\r\n"
    @connection.flush
    result = @connection.gets
    if result.start_with?('$')
      value = ''
      bytesize = result.tap { |r| r[0] = '' }.to_i
      return nil if bytesize < 0 # Key does not exist
      @connection.read bytesize + 2, value
      value.chomp! CRLF
      value
    else
      raise "idk what to do with this: #{result.inspect}"
    end
  end
end

gemfile true do
  source 'https://rubygems.org'
  gem 'redis'
  gem 'hiredis'
  gem 'benchmark-ips'
end

require 'benchmark'
require 'benchmark/ips'
require 'redis'
require 'hiredis'

redis = Redis.new(driver: :ruby)
hiredis = Redis.new(driver: :hiredis)
myredis = MyRedis.new
key = "foo"
cpu_time_iterations = 100_000

# Quick functionality check on our custom Redis
redis.set key, "test"
if (myredis_value = myredis.get(key)) == (redis_value = redis.get(key))
  puts "Cache-hit function check passed"
else
  raise "Value mismatch. Redis: #{redis_value.inspect}, MyRedis: #{myredis_value.inspect}"
end
redis.del key
if (myredis_value = myredis.get(key)).nil?
  puts "Cache-miss function check passed"
else
  raise "MyRedis cache miss broken: #{myredis_value.inspect}"
end


puts
puts "Cache hit (returns a 2KB string)"
redis.set key, "." * 2048 # 2KB cache entry
Benchmark.ips do |x|
  x.report("ruby") { redis.get key }
  x.report("hiredis") { hiredis.get key }
  x.report("myredis") { myredis.get key  }

  x.compare!
end

puts
puts "Cache hit CPU time"
puts "ruby   : %.06f" % [ruby_cpu = Benchmark.measure { cpu_time_iterations.times { redis.get key } }.total]
puts "hiredis: %.06f" % [hiredis_cpu = Benchmark.measure { cpu_time_iterations.times { hiredis.get key } }.total]
puts "myredis: %.06f" % [myredis_cpu = Benchmark.measure { cpu_time_iterations.times { myredis.get key } }.total]
fastest = [ruby_cpu, hiredis_cpu, myredis_cpu].min
puts "\e[3Aruby   : %.06f (%.02fx)" % [ruby_cpu, ruby_cpu / fastest]
puts "hiredis: %.06f (%.02fx)" % [hiredis_cpu, hiredis_cpu / fastest]
puts "myredis: %.06f (%.02fx)" % [myredis_cpu, myredis_cpu / fastest]

puts
puts "Cache miss (returns nil)"
redis.del key
Benchmark.ips do |x|
  x.report("ruby") { redis.get key }
  x.report("hiredis") { hiredis.get key }
  x.report("myredis") { myredis.get key  }

  x.compare!
end

puts
puts "Cache miss CPU time"
puts "ruby   : %.06f" % [ruby_cpu = Benchmark.measure { cpu_time_iterations.times { redis.get key } }.total]
puts "hiredis: %.06f" % [hiredis_cpu = Benchmark.measure { cpu_time_iterations.times { hiredis.get key } }.total]
puts "myredis: %.06f" % [myredis_cpu = Benchmark.measure { cpu_time_iterations.times { myredis.get key } }.total]
fastest = [ruby_cpu, hiredis_cpu, myredis_cpu].min
puts "\e[3Aruby   : %.06f (%.02fx)" % [ruby_cpu, ruby_cpu / fastest]
puts "hiredis: %.06f (%.02fx)" % [hiredis_cpu, hiredis_cpu / fastest]
puts "myredis: %.06f (%.02fx)" % [myredis_cpu, myredis_cpu / fastest]