Sidekiq's bottlenecks

sss-sidekiq-bottlenecks.md

Sidekiq and background jobs

Getting started

In our gemfile, Sidekiq is configured to be at the edge of version 6, which is compatible with our server's Redis version. Background jobs are trackable via Sidekiq's dashboard, accessible here (for admins only, authentication is required).

Understanding Sidekiq and its challenges

This is the list of all the background jobs that manipulate our app's data. Most of them work on demand, some are triggered by CRON jobs set in Heroku.

Name	Role/functionb	Data direction	Frequency
ApplicationUpdate	Checks for changes in application status in Bullhorn (information only)	To Rails app	On demand
ApplicationUpdates	Checks for changes in application status in Bullhorn and applies them in the rails app	To Rails app	Every 30 minutes
CheckAppDealt	Unused, method not defined	To Rails app	On demand
CheckAppUpdate	Checks for changes in application status in Bullhorn and applies them in the rails app	To Rails app	On demand
CoupleApplicationUpload	Uploads user application to Bullhorn	To BH	On demand
CredentialRefreshJob	Fetches Bullhorn tokens and updates the app credentials	To Rails app	On demand / Every 4 minutes
DailyReportGenerator	Generates an internal report	To Rails app	On demand
DestroyAct	Destroys Activity Log - internal	In Rails app	On demand
DestroyApp	Deletes a candidate's application	In Rails app	On demand
GoogleIndexingJob	Notifies Google when job pages are added or removed	To Google	On demand
JobApplicationUpload	Uploads user application to Bullhorn	To BH	On demand
JobLoader	Fetches an existing job's information on Bullhorn	To Rails app	On demand
NewJobLoaderJob	Fetches a new job's information on Bullhorn	To Rails app	Every 20 minutes
RejectApplication	Closes application on Bullhorn	To BH	On demand
UpdateOrCreateJob	Creates a Job on the rails app with provided information	In Rails app	On demand
UpdateRegionsJob	Retrieve job and update it region if Bullhorn region is different from app's region	To Rails app	On demand
UploadMissingCandidatesJob	If user has no bullhorn_id, upload it to Bullhorn	To BH	On demand
UserCvUploadJob	Uploads a CV to Bullhorn	To BH	On demand
UserRoleJob	Updates user desired roles	In Rails app	On demand
UserUploadJob	Uploads a user to Bullhorn	To BH	On demand

Sidekiq and Bullhorn

Several background jobs rely on interactions with Bullhorn's API. For them to work, it's important that SilverSwan remains logged in to Bullhorn via the silverswanrecruitment.api credentials accessible on Basecamp. When connection fails because the Bullhorn's API tokens are expired, a refresh is necessary. If the background jobs can no longer reach Bullhorn's API, they will start to pile up in the queue. The app must be reconnected via this link, a manual log in to Bullhorn might be required when accessing it.

Main Sidekiq bottlenecks : UserUpload job

Every 20 minutes, a CRON job triggers the UploadMissingCandidates background job on our server. In turn, this process triggers a parallel action : upload_candidate_to_bullhorn. This user method triggers a background job by doing UserUploadJob*.perform_later(id) that can get stuck in Sidekiq's queue. Let's see how this works.

The UserUpload background job, is simple : when a new user appears on the app, UploadMissingCandidates will tell UserUpload to ask Bullhorn if it exsists on the Bullhorn database by doing :

bh_candidate_search = api_get("/search/Candidate?query=email:#{user.email}")

This returns a certain total of candidates. If the total is not greater than zero (response : {"total"=>0, "start"=>0, "count"=>0, "data"=>[]}), then it assumes does not exist yet on Bullhorn so it's safe to create it! So it runs :

response = api_put("/entity/Candidate", to_upload)
user.update_attribute(:bullhorn_id, response['changedEntityId'])

The first line of code generates a PUT request to the Bullhorn API that creates a user withe the information stored in the to_upload variable. The second line of code assigns the user its retrieved bullhorn_id given by response['changedEntityId']) .

The problem is that when we do a PUT request to create said candidate on Bullhorn, sometimes the API returns :

=> {"errorMessage"=>"error persisting an entity of type: Candidate", "errorMessageKey"=>"errors.cannotPersistEntity", "errorCode"=>500, "errors"=>[{"detailMessage"=>"", "propertyName"=>"", "severity"=>"ERROR", "type"=>"UNKNOWN_INTERNAL_ERROR"}], "entityName"=>"Candidate"}

That is because the following fields must be formatted correctly :

    to_upload = {
      firstName: user.first_name, //String
      lastName: user.last_name, //String
      name: user.full_name, //String
      gender: user.gender_initial, //String
      phone: user.phone_number, //String
      email: user.email, //String
      dateOfBirth: user.dob.to_time.to_i * 1000, //Integer
      experience: user.experience, //Integer
      source: ["Silver Swan Search"], //Array 
      customTextBlock10: user.bio, //String
      customText15: user.nationality.truncate(100),
      customText13: user.fluent_languages.map(&:language).join(','), //String
      customText18: user.conversational_languages.map(&:language).join(','), //String
      customText17: user.source //String
    }

Solution

• Enforce strict validations on user model
• Constraints applied on the onboarding/sign_up process
• Constraints applied on the profile update/edit pages

Killing a job that is stuck on the queue

Sometimes, jobs can get stuck on a loop in Sidekiq's queue. To kill them:

1. Make sure the data is persisted
2. Execute this code in the servers console heroku run rails c --app=silver-swan-search

queue = Sidekiq::Queue.new("default")

queue.each do |job|
  argument = job.args.first['arguments']
  if argument = 'the_number_youre_after'
    job.delete
    puts "Job with argument #{argument} deleted"
  end
end

Identifying the top 10 repeated jobs and killing them

job_frequency = Hash.new(0)
job_types = {}

queue.each do |job|
  argument = job.args.first['arguments']
  job_type = job.args.first['job_class']
  job_frequency[argument.first] += 1
  job_types[argument.first] = job_type
end

top_10_jobs = job_frequency.sort_by { |_k, v| -v }.first(10)
delete_list = []
top_10_jobs.each do |job|
  delete_list << job[0]
end

delete_list.each do |int|
  queue.each do |job|
    argument = job.args.first['arguments']
    if argument.include?(int)
      job.delete
      puts "Job with argument #{argument} deleted"
    end
  end
end