Sending emails with ActionMailer and Sidekiq

00.md

Sending emails with ActionMailer and Sidekiq

Send email asynchroniously using Sidekiq.

ActionMailer

Create your mailer us usual:

# app/mailers/users_mailer.rb

class UsersMailer < ActionMailer::Base

def welcome_email(user_id)
    @user = User.find(user_id)

    mail(   :to      => @user.email,
            :subject => "Welcome"
    ) do |format|
      format.text
      format.html
    end
  end
end

Views for email:

app/views/users_mailer/welcome_email.html.erb - HTML version
app/views/users_mailer/welcome_email.text.erb - TEXT version

Send email:

user = User.find(1)

mail = UsersMailer.welcome_email(user.id)
#mail.deliver_now
mail.deliver_later

    

Sidekiq

Gemfile:

gem 'sidekiq'

Install Redis

Redis provides data storage for Sidekiq. It holds all the job data along with runtime and historical data

Configure Sidekiq

To make #deliver_later work we need to tell ActiveJob to use Sidekiq. As long as Active Job is setup to use Sidekiq we can use #deliver_later.

# config/initializers/active_job.rb
  
  config.active_job.queue_adapter = :sidekiq

Environment file:

# config/environments/development.rb

Rails.application.configure do
  ...
  
  config.active_job.queue_adapter = :sidekiq
  
  config.action_mailer.perform_deliveries = true
  config.action_mailer.raise_delivery_errors = true

  config.action_mailer.delivery_method = :smtp
  
  config.action_mailer.smtp_settings = { ... }
  

end

Read more about ActionJob and Sidekiq: https://github.com/mperham/sidekiq/wiki/Active-Job

Configure Sidekiq

config/sidekiq.yml:

---
:concurrency: 1
:queues:
  - default
  - mailers

Specify Redis namespace for different environments:

# config/initializers/sidekiq.rb

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://localhost:6379/0', namespace: "app3_sidekiq_#{Rails.env}" }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://localhost:6379/0', namespace: "app3_sidekiq_#{Rails.env}" }
end

Sidekiq and ActionMailer (ActionJob)

By default, jobs to deliver emails will be placed in queue named :mailers. To change queue name for ActionMailer use this config:

# config/environments/development.rb

  config.active_job.queue_adapter = :sidekiq

  config.active_job.queue_name_prefix = "mysite"
  config.active_job.queue_name_delimiter = "_"

This will use queues named :mysite_mailers, etc.

!!! important. You may need to include new queue names in sidekiq.yml file:

# config/sidekiq.yml

---
:concurrency: 1
:queues:
  - default
  - mailers

  - mysite_default
  - mysite_mailers

Sidekiq and Devise

Read this: https://github.com/mperham/sidekiq/wiki/Devise

Run Sidekiq

bundle exec sidekiq --environment development -C config/sidekiq.yml 

God + Sidekiq

Use god for monitoring and running sidekiq automatically: https://gist.github.com/maxivak/05847dc7f558d5ef282e

RSpec tests

RSpec tests for ActionMailer

In these tests we do not use Sidekiq.

test environment:

# config/environments/test.rb

Rails.application.configure do
 ...
 
  config.active_job.queue_adapter = :test

  config.action_mailer.perform_deliveries = true
  config.action_mailer.delivery_method = :test
  config.action_mailer.raise_delivery_errors = true

 
 

Test that deliver_later method was called:

user = User.first

    #
    message_delivery = instance_double(ActionMailer::MessageDelivery)
    expect(UsersMailer).to receive(:welcome_email).with(user.id).and_return(message_delivery)
    expect(message_delivery).to receive(:deliver_later)
    

#   
mail = UsersMailer.welcome_email(user.email)
mail.deliver_later
    
    

RSpec tests for Sidekiq

test environment:

# config/environments/test.rb

Rails.application.configure do
 ...
   
  config.active_job.queue_adapter = :sidekiq

  config.action_mailer.perform_deliveries = true
  config.action_mailer.delivery_method = :test
  config.action_mailer.raise_delivery_errors = true

 

spec/rails_helper.rb:

# sidekiq
require 'sidekiq/testing'
Sidekiq::Testing.fake!  # by default it is fake

User Sidekiq::Worker.jobs.size to see the number of jobs in the queue.

Test that email was enqueued:

RSpec.describe "Test sending email with sidekiq", :type => :request do

  it 'send email to sidekiq' do

    user = User.first
      
    expect{
      UsersMailer.welcome_email(user.id).deliver_later
    }.to change( Sidekiq::Worker.jobs, :size ).by(1)

  end

end

@SeriusDavid deliver_now http://api.rubyonrails.org/classes/ActionMailer/MessageDelivery.html#method-i-deliver_now

@r4mbo7 not possible with active_job, you would have to create a sidekiq job to be able to use the sidekiq advanced options like retry etc.

👍

This guide is extremely helpful, thank you!

I wanted to mention an issue I had when testing if the email was enqueued with Sidekick - in case it helps anyone else. Using argument syntax with change rather than block syntax wasn't working for me. The value of .size didn't change.

RSpec.describe "Test sending email with sidekiq", :type => :request do
  it 'send email to sidekiq' do
    user = User.first
      
    expect{
      UsersMailer.welcome_email(user.id).deliver_later
    }.to change(Sidekiq::Worker.jobs, :size).by(1) # `size` returns `0`
  end
end

I believe the reason is that with argument syntax, Sidekiq::Worker.jobs is evaluated before the expect runs, and it returns a different array instance each time it's executed.

So checking .size in this case was returning the size of the jobs instance before the expect block executes.

Changing to a block syntax so that .jobs is evaluated both before and after the expect block solved my issue:

RSpec.describe "Test sending email with sidekiq", :type => :request do
  it 'send email to sidekiq' do
    user = User.first
      
    expect{
      UsersMailer.welcome_email(user.id).deliver_later
    }.to change { Sidekiq::Worker.jobs.size }.by(1) # `size` returns `1`
  end
end

Is there anyway to make sidekiq retry sending the email 5 times before throwing an exception? - I currently get 2-3 exceptions per day that are:

Net::ReadTimeoutActionMailer::MailDeliveryJob@default
gems/net-protocol-0.1.3/lib/net/protocol.rb:219 rbuf_fill

They seem harmless and happen when Google servers are busy, a retry fixes it but they are clocking up Bugsnag.

Great guide!

I had an issue where the jobs were always getting queued in the :default queue and not :mailers. I'm not sure why, but I had to explicitly set the mailer queue, even though ActionMailer defaults it internally to :mailers

# application.rb
config.action_mailer.deliver_later_queue_name = :mailers

Secondly, I also wanted an easier way to test which mailers were enqueued in RSpec tests. I use Sidekiq + RSpec and came up with this helper method. The description is probably a bit too wordy, but I wanted to be explicit about what it's doing.

This just returns a hash of mailers that are enqueued so you can easily access the mailer name and args.

usage:

expect do
  Devise::Mailer.confirmation_instructions(User.last, 'abcdefg', {}).deliver_later
end.to change { enqueued_mailers.count }.by(1)

email = enqueued_mailers.last
expect(email[:klass]).to eq(Devise::Mailer)
expect(email[:mailer_name]).to eq(:confirmation_instructions)
expect(email[:args][:record]).to eq(User.last)

Method source:

# When a mailer is enqueued with `#deliver_later`, it generates an
# ActionMailer job (which implements the ActiveJob interface).
#
# To be able to enqueue this job onto Sidekiq, it has to be further wrapper
# in a Sidekiq Job Wrapper that will enqueue it with keys that sidekiq
# expects, like `jid`, `retry`, etc...
#
# For example, here's the result of calling
#
#   > Devise::Mailer.confirmation_instructions(
#       User.find(108),
#       'pU8s2syM1pYN523Ap2ix',
#       {}
#     ).deliver_later
#
#   > Sidekiq::Worker.jobs
#   => [
#     {
#       "class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper",
#       "wrapped"=>"ActionMailer::MailDeliveryJob",
#       "queue"=>"mailers",
#       "args"=>
#         [{
#           "job_class"=>"ActionMailer::MailDeliveryJob",
#           "job_id"=>"17464385-ed14-4490-ab10-a0770870c169",
#           "provider_job_id"=>nil,
#           "queue_name"=>"mailers",
#           "priority"=>nil,
#           "arguments"=>[
#             "Devise::Mailer",
#             "confirmation_instructions",
#             "deliver_now",
#             {
#               "args"=>[
#                 {"_aj_globalid"=>"gid://familyties/User/108"},
#                 "pU8s2syM1pYN523Ap2ix",
#                 {"_aj_symbol_keys"=>[]}
#               ],
#               "_aj_ruby2_keywords"=>["args"]
#             }
#           ],
#           "executions"=>0,
#           "exception_executions"=>{},
#           "locale"=>"en",
#           "timezone"=>"UTC",
#           "enqueued_at"=>"2023-05-31T15:31:34Z"
#         }],
#       "retry"=>true,
#       "jid"=>"0c6ebfceee4cddc9ccd557b4",
#       "created_at"=>1685547094.2727168,
#       "enqueued_at"=>1685547094.2728298
#     }
#   ]
#
# This nested structure can be inconvenient for testing, so the below
# method deserializes this information and produces a simple hash that
# can be used for testing
#
#   expect do
#     post :create, params: params
#   end.to change { enqueued_mailers.count }.by(1)
#
#   email = enqueued_mailers.last
#   expect(email[:klass]).to eq(Devise::Mailer)
#   expect(email[:mailer_name]).to eq(:confirmation_instructions)
#   expect(email[:args][:record]).to eq(User.last)
def enqueued_mailers
  Sidekiq::Worker.jobs.map do |job|
    # `Sidekiq:Worker.jobs` returns all jobs. We only want to filter on those
    # in our action_mailer queue
    queue_name =
      Rails.application.config.action_mailer.deliver_later_queue_name
    next unless job['queue'] == queue_name.to_s

    mailer_klass, mailer_name, _method_name, args =
      ActiveJob::Arguments.deserialize(job['args'][0]['arguments'])

    mailer_klass = mailer_klass.constantize
    params =
      mailer_klass.instance_method(mailer_name).parameters.map { |p| p[1] }

    args = args[:args]

    enqueued_at = Time.zone.at(job['enqueued_at']) if job['enqueued_at']
    at = Time.zone.at(job['at']) if job['at']

    {
      klass: mailer_klass,
      mailer_name: mailer_name.to_sym,
      args: Hash[params.zip(args)],
      enqueued_at: enqueued_at,
      at: at
    }
  end.compact
end

To get proper error handling and control over retries, I ended up doing this:

class MyMailerWorker
  include Sidekiq::Worker
  sidekiq_options queue: :mailer, retry: 2, backtrace: true

  def perform(location_id, event_key, name = nil, event_id = nil)
    # Setup code
    begin
      MyMailer.welcome(params).deliver_now
    rescue StandardError => e
      # Error handling code
    end
  end
end

Not sure if there is a better solution.

Thanks for this great reference.

Note that redis-namespace is no longer supported. You might want to update that section.

Sidekiq.configure_client do |config|
  config.redis = { db: 1 }
end