Benchmarking PR #17227 to Rails

Analysis.md

Performance comparison: throw(:abort) vs. raise ActiveSuppot::CallbackAborted #################################################################################

In the current Rails stable (4.2) and master (5.0.0.alpha), callbacks can be halted with return false.

PR #17227 replaces this behavior with an explicit throw(:abort).

In a comment, @tenderlove asks whether this will have a significant impact on the performance, and whether it would be better (performance-wise) to use a raise/rescue construct instead.

The code below can be used to benchmark the impact of these different strategies on ActiveRecord callbacks. Notice that PR #17227 also changes ActiveModel and ActiveSupport callback, but this benchmark only focuses on ActiveRecord.

Experiment

I created a scaffold model Post with a string field Title. Then I benchmark the code to create a new Post under different sets of callbacks:

• Post has no callbacks at all
• PostWithPassingCallbacks has callbacks that always return true
• PostWithFalseCallback has one callback that returns false
• PostWithThrowingCallback has one callback that calls throw :abort
• PostWithRaisingCallback has one callback that calls raise ActiveSupport::CallbackAborted

Results

These are my observations:

• [Comparing z1 with z2] The current 5.0.0.alpha is already faster than 4.2.0 (stable) under every condition. This is not related to PR #17227 but good to know.
• [Comparing z2 with z3/z4] Replacing return false with a different strategy (either throw/catch or raise/rescue) slightly decreases the performance in some conditions and slightly increases the performance in other conditions.
• [Comparing z4 with z3] Raising/rescuing an error rather than throwing/catching an exception does not have a significantly positive impact on performance

According to these results, my opinion is to keep PR #17227 as it is, using throw(:abort) rather than raise ActiveSupport::CallbackAborted. This favors readability of code as well.

Gemfile

source 'https://rubygems.org'

gem 'benchmark-ips'
gem 'sqlite3'

# z1. To benchmark stable Rails (which halts callbacks on false), uncomment:
#
# gem 'rails', '4.2.0'

# z2. To benchmark today's master Rails (which halts callbacks on false), uncomment:
#
# gem 'rails', github: 'rails/rails', ref: 'd08dc37'

# z3. To benchmark PR #17227 (which halts callbacks on `throw :abort`), uncomment:
#
# gem 'rails', github: 'claudiob/rails', branch: 'explicitly-abort-callbacks'
# gem 'arel', github: 'rails/arel'

# z4. To benchmark an alternative to PR #17227 (which halts callbacks on `raise ActiveSupport::CallbackAborted`), uncomment:
# 
# gem 'rails', github: 'claudiob/rails', branch: 'explicitly-raise-callbacks'
# gem 'arel', github: 'rails/arel'

test.rb

require 'active_record'
require 'benchmark/ips'
require 'benchmark'

ActiveRecord::Base.establish_connection(
  adapter: 'sqlite3',
  host: 'localhost',
  database: 'db/development.sqlite3',
  pool: 5,
  timeout: 5000
)

# if you don't have a table in your db, create it like this:
#
# ActiveRecord::Schema.define do
#   create_table :posts do |t|
#     t.string :title
#   end
# end

class Post < ActiveRecord::Base
end

class PostWithPassingCallbacks < Post
  before_validation {true}
  before_save       {true}
  before_create     {true}
end

class PostWithFalseCallback < Post
  before_validation {true}
  before_save       {true}
  before_create     {false}
end

class PostWithThrowingCallback < Post
  before_validation {true}
  before_save       {true}
  before_create     {throw :abort}
end

class PostWithRaisingCallback < Post
  before_validation {true}
  before_save       {true}
  before_create     {raise ActiveSupport::CallbackAborted}
end

class TestIfAbortIsCatched < Post
  before_save     {throw :abort}
end

can_catch_abort = false
catch(:abort) do
  TestIfAbortIsCatched.new.save
  can_catch_abort = true
end

############################################################

Benchmark.ips do |r|
  r.report('without callbacks ') do
    Post.create title: 'title'
  end

  r.report('with passing callbacks ') do
    PostWithPassingCallbacks.create title: 'title'
  end

  r.report('with callback returning false ') do
    PostWithFalseCallback.create title: 'title'
  end

  r.report('with callback throwing exception') do
    catch(:abort) do
      PostWithThrowingCallback.create title: 'title'
    end
  end if can_catch_abort

  r.report('with callback raising error') do
    PostWithRaisingCallback.create title: 'title'
  end if defined? ActiveSupport::CallbackAborted
end

z1. Benchmark ActiveRecord 4.2.0

$ bundle exec ruby test.rb
Calculating -------------------------------------
  without callbacks               83.000  i/100ms
with passing callbacks            76.000  i/100ms
with failing callbacks           279.000  i/100ms

-------------------------------------------------
  without callbacks              839.881  (± 7.4%) i/s -      4.233k
with passing callbacks           776.129  (± 6.7%) i/s -      3.876k
with failing callbacks             2.822k (± 4.4%) i/s -     14.229k

z2. Benchmark ActiveRecord 5.0.0.alpha

$ bundle exec ruby test.rb
Calculating -------------------------------------
  without callbacks               91.000  i/100ms
with passing callbacks            81.000  i/100ms
with failing callbacks           345.000  i/100ms
-------------------------------------------------
  without callbacks              944.907  (± 7.8%) i/s -      4.732k
with passing callbacks           827.360  (± 8.2%) i/s -      4.131k
with failing callbacks             3.503k (± 6.3%) i/s -     17.595k

z3. Benchmark PR #17227

$ bundle exec ruby test.rb
Calculating -------------------------------------
  without callbacks               93.000  i/100ms
with passing callbacks            81.000  i/100ms
with failing callbacks           215.000  i/100ms
with failing callbacks new style 299.000  i/100ms
-------------------------------------------------
  without callbacks              937.414  (± 8.9%) i/s -      4.650k
with passing callbacks           828.052  (± 7.4%) i/s -      4.131k
with failing callbacks             2.213k (± 3.0%) i/s -     11.180k
with failing callbacks new style   3.512k (± 4.8%) i/s -     17.641k

z4. Benchmark alternative to PR #17227

$ bundle exec ruby test.rb
Calculating -------------------------------------
  without callbacks               90.000  i/100ms
with passing callbacks            81.000  i/100ms
with failing callbacks           221.000  i/100ms
with failing callbacks new style 327.000  i/100ms
-------------------------------------------------
  without callbacks              921.979  (± 6.5%) i/s -      4.590k
with passing callbacks           812.643  (± 6.2%) i/s -      4.050k
with failing callbacks             2.228k (± 2.7%) i/s -     11.271k
with failing callbacks new style   3.272k (± 4.6%) i/s -     16.350k