- What is a background worker and when do we want to use one
- How to implement a basic background worker with Sidekiq
When building websites, it’s important to keep your response times down. Long-running requests tie up server resources, degrade user perception of your site, and make it hard to manage failures.
There’s a solution to this: return a successful response, and then schedule some computation to happen later, outside the original request/response cycle.
We do this with background workers. A background worker allows you to run process on a separate dedicated thread. We can move tasks that don't have to happen in real time to their own process so that the app can move on, rather than causing a backlog of requests which slows everything down.
How do you identify areas of the application that can benefit from a background job? Some common areas for asynchronous work:
- Data Processing - e.g. generating thumbnails or resizing images
- 3rd Party APIs - interacting with a service outside of your site
- Maintenance - expiring old sessions, sweeping caches
- Email - a request that causes an email to be sent
Applications with good OO design make it easy to send jobs to workers, poor OO makes it hard to extract jobs since responsibilities tend to overlap.
Let's look at an example of backgrounding a task using Sidekiq. We'll use the "Congress Tracker" application for the following demo. The setup branch is called "sidekiq_setup" Clone and bundle it if you do not already have it from earlier this mod:
git clone git@github.com:turingschool-examples/congress-tracker-2108.git
cd congress-tracker-2108
bundle
rake db:{create,migrate}
rails s
We've added some code that takes an email and a random thought to generate an email. Start the server rails s and you should be able to view the app at http://localhost:3000 which has form inputs for an email address and a thought. When you submit that form you should feel the pain of a slow page load.
Why?
Check out our UserNotifierMailer mailer. A 5 second delay has been hard-coded in to simulate a real-life delay.
Notice that this process takes a very long time. What we have here is a perfect candidate for a background process:
- Operation is slow
- User's interaction with the process is already asynchronous (submit the form then go check their email)
- Operation is well-encapsulated behind the UserNotifier interface
- Operation requires relatively little data as inputs (email address and random thought).
Sidekiq and Resque are the 2 most popular queuing libraries for Ruby. For this application, we'll use Sidekiq.
Sidekiq uses Redis as a memory store for enqueued jobs, so first make sure you have redis installed and running.
Run redis-server
If you don't already have redis, install it with homebrew:
brew update && brew install redis
Then run redis-server.
This command starts the redis server on port 6379. Once the redis server is running it is ready to queue jobs that still need to be done.
You can check if your redis process is running by executing the command
redis-cli:
$ redis-cli
127.0.0.1:6379>
The need for running our Redis server now is similar to how Postgres must be running in order for ActiveRecord to interact with our database.
Sidekiq is the tool we will use to manage our background workers.
We can add the sidekiq gem to our Gemfile:
gem 'sidekiq'
bundle the app and you should now be able to run a sidekiq process by executing the command:
bundle exec sidekiq
You should get the fancy Sidekiq ASCII logo in your terminal:
s
ss
sss sss ss
s sss s ssss sss ____ _ _ _ _
s sssss ssss / ___|(_) __| | ___| | _(_) __ _
s sss \___ \| |/ _` |/ _ \ |/ / |/ _` |
s sssss s ___) | | (_| | __/ <| | (_| |
ss s s |____/|_|\__,_|\___|_|\_\_|\__, |
s s s |_|
s s
sss
sss
Next we need to add a Sidekiq initializer:
Add sidekiq.rb to your initializers folder:
config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://localhost:6379/0' }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://localhost:6379/0' }
endIf your server is running, you must restart it after adding an initializer for it to take effect.
Sidekiq also comes with a handy dashboard UI so we can see what our jobs are doing. We can set this up by adding the following to our routes file:
require 'sidekiq/web'
Rails.application.routes.draw do
mount Sidekiq::Web => "/sidekiq"
endWe do not need to do this for the Congress Tracker app, however, if you are running an API only application (like your consultancy backend apps) you must enable some sessions middleware that is excluded in API only apps by default:
In config/application.rb add the following to the application class:
config.session_store :cookie_store, key: '_interslice_session'
config.middleware.use ActionDispatch::Cookies
config.middleware.use config.session_store, config.session_optionsNow we have sidekiq running with our application, but so far it doesn't do anything for us. Let's create a worker to handle our email.
In the terminal:
rails generate sidekiq:worker gif_sender
This will generate your worker class within a directory called workers:
class GifSenderWorker
include Sidekiq::Worker
def perform(*args)
# Do something
end
endIt is sidekiq convention to use the suffix of _Worker for your worker classes.
Within a Sidekiq job, the instance method #perform is what gets
called whenever a job appears for our worker to do. Other methods can live within the job class, but #perform is what will be invoked when running the worker.
Let's think about what needs to go in here and about what inputs are required for the worker to do its job:
- Needs to take in the email address and thought (since these are the parameters needed to send the email)
- Needs to send an email using the
UserNotifier
Given these constraints, it might look something like:
class GifSenderWorker
include Sidekiq::Worker
def perform(email, thought)
UserNotifierMailer.send_randomness_email(email, thought).deliver_now
end
endThe Sidekiq worker defines what actual work will be done whenever our background process is invoked. Now we just need to actually invoke it.
With Sidekiq, we dispatch a job for a worker to do later by
calling .perform_async on our worker and
providing it whatever arguments are needed for the job.
Under the hood, the .perform_async method writes data into
Redis indicating the type of job which needs to be done
and the data associated with it. The workers (in a separate
process) are monitoring the queue so that whenever new jobs
appear, they can spring into action and do them!
Also commonly used is .perform_at which can be reviewed in the sidekiq docs. This method allows you to specify the date/time a job should execute.
But enough chit-chat, let's see what it looks like to actually
queue the job. Recall that we were previously sending the
email directly from our MailersController. Let's replace
the line that was sending the email with this line to
queue our job instead:
GifSenderWorker.perform_async(params[:mailers][:email], params[:mailers][:thought])Remember -- the arguments passed in to the .perform_async method here
will eventually be handed to your worker's #perform method, so make
sure they match up.
Let's try this out now - hopefully that painful delay is no more!
Sidekiq provides a dashboard for us to monitor our local job queues.
In our routes file, we'll need to mount the Sidekiq dashboard:
require 'sidekiq/web'
mount Sidekiq::Web => '/sidekiq'Now you can navigate to http://localhost:3000/sidekiq/. This dashboard is very useful for testing out jobs and receiving confirmation that everything is queued according to plan.
What is the difference between a worker and a job?
- What is a background worker? Why would you use a background worker?
- What is Sidekiq?
- What is Redis?
- How do Sidekiq, Redis, and Rails interact? Draw a diagram?