Sidekiq gracefull startup and shutdown in Kubernetes

README.md

Sidekiq on Kubernetes

Docs

• Sidekiq Wiki: Deployment
• Kubernetes best practices: terminating with grace
• Configure Liveness, Readiness and Startup Probes

Startup

Timeout

Timeout option is an important option, that used to identify how long sidekiq should wait while the job should be force terminated.

Scenarios

Timeout is less than JOB duration

In one terminal run sidekiq process with timeout 5 seconds:

bundle exec sidekiq -r ./main.rb -t 5 -c 2

open another terminal and add Job for 10 seconds work

bundle exec ruby add_job.rb 10
sleep 1
pkill -TERM sidekiq

After rerturn to the sidekiq terminal you see error message in logs and job returned to Redis and mark job as failed.

2021-05-21T19:10:13.746Z pid=28238 tid=ksy INFO: Got TERM, shutting down process...
2021-05-21T19:10:13.848Z pid=28238 tid=ksy INFO: Pausing to allow workers to finish...
2021-05-21T19:10:18.735Z pid=28238 tid=ksy WARN: Terminating 1 busy worker threads
2021-05-21T19:10:18.735Z pid=28238 tid=ksy WARN: Work still in progress [#<struct Sidekiq::BasicFetch::UnitOfWork queue="queue:default", job="{\"class\":\"HardWorker\",\"queue\":\"default\",\"args\":[10],\"retry\":true,\"jid\":\"3b12681779811a92975a8f9d\",\"created_at\":1621624210.840917,\"enqueued_at\":1621624210.840918}">]
2021-05-21T19:10:18.735Z pid=28238 tid=ksy INFO: Pushed 1 jobs back to Redis
2021-05-21T19:10:18.736Z pid=28238 tid=kxe class=HardWorker jid=3b12681779811a92975a8f9d elapsed=7.893 INFO: fail

Timeout is higher than JOB duration

In one terminal run sidekiq process with timeout 30 seconds:

bundle exec sidekiq -r ./main.rb -t 30 -c 2

open another terminal and add Job for 5 seconds work

bundle exec ruby add_job.rb 5
sleep 1
pkill -TERM sidekiq

Sidekiq Logs:

2021-05-21T19:12:13.497Z pid=28332 tid=ku4 INFO: Scheduler exiting...
2021-05-21T19:12:13.497Z pid=28332 tid=kvk INFO: Got TERM, shutting down process...
2021-05-21T19:12:13.603Z pid=28332 tid=kvk INFO: Pausing to allow workers to finish...
2021-05-21T19:12:20.035Z pid=28332 tid=kr4 class=HardWorker jid=ec09fea6f0915b96cff449d9 INFO: HardWorker: Finished
2021-05-21T19:12:20.035Z pid=28332 tid=kr4 class=HardWorker jid=ec09fea6f0915b96cff449d9 elapsed=10.004 INFO: done

Run multiple jobs

In one terminal run sidekiq process with timeout 30 seconds:

bundle exec sidekiq -r ./main.rb -t 30 -c 2

open another terminal and add Job for 5 seconds work

bundle exec ruby add_job.rb 10
bundle exec ruby add_job.rb 15
sleep 1
pkill -TERM sidekiq

Sidekiq Logs:

2021-05-21T19:20:43.749Z pid=28488 tid=koc class=HardWorker jid=52620a57082ee28add5c7629 INFO: start
2021-05-21T19:20:43.749Z pid=28488 tid=koc class=HardWorker jid=52620a57082ee28add5c7629 INFO: HardWorker: sleep for 10
2021-05-21T19:20:44.001Z pid=28488 tid=kqc class=HardWorker jid=02da3ddebc1cd837a1832d23 INFO: start
2021-05-21T19:20:44.001Z pid=28488 tid=kqc class=HardWorker jid=02da3ddebc1cd837a1832d23 INFO: HardWorker: sleep for 15
2021-05-21T19:20:45.045Z pid=28488 tid=klw INFO: Shutting down
2021-05-21T19:20:45.045Z pid=28488 tid=klw INFO: Terminating quiet workers
2021-05-21T19:20:45.045Z pid=28488 tid=klw INFO: Got TSTP, stopping further job processing...
2021-05-21T19:20:45.045Z pid=28488 tid=kog INFO: Scheduler exiting...
2021-05-21T19:20:45.046Z pid=28488 tid=klw INFO: Got TERM, shutting down process...
2021-05-21T19:20:45.151Z pid=28488 tid=klw INFO: Pausing to allow workers to finish...
2021-05-21T19:20:53.754Z pid=28488 tid=koc class=HardWorker jid=52620a57082ee28add5c7629 INFO: HardWorker: Finished
2021-05-21T19:20:53.754Z pid=28488 tid=koc class=HardWorker jid=52620a57082ee28add5c7629 elapsed=10.005 INFO: done
2021-05-21T19:20:59.006Z pid=28488 tid=kqc class=HardWorker jid=02da3ddebc1cd837a1832d23 INFO: HardWorker: Finished
2021-05-21T19:20:59.007Z pid=28488 tid=kqc class=HardWorker jid=02da3ddebc1cd837a1832d23 elapsed=15.005 INFO: done

It waits for 2 jobs and the only exited

During the termination don't take another jobs

In one terminal run sidekiq process with timeout 30 seconds:

bundle exec sidekiq -r ./main.rb -t 30 -c 2

open another terminal and add Job for 20 seconds work and send signal to stop sidekiq and try to add another job

bundle exec ruby add_job.rb 20
sleep 1
pkill -TERM sidekiq
bundle exec ruby add_job.rb 5

Sidekiq logs:

2021-05-21T19:18:40.190Z pid=28417 tid=kot class=HardWorker jid=36a961b48d008ed18e6098f5 INFO: start
2021-05-21T19:18:40.190Z pid=28417 tid=kot class=HardWorker jid=36a961b48d008ed18e6098f5 INFO: HardWorker: sleep for 20
2021-05-21T19:18:41.226Z pid=28417 tid=kkd INFO: Shutting down
2021-05-21T19:18:41.226Z pid=28417 tid=kkd INFO: Terminating quiet workers
2021-05-21T19:18:41.226Z pid=28417 tid=kkd INFO: Got TSTP, stopping further job processing...
2021-05-21T19:18:41.226Z pid=28417 tid=kpd INFO: Scheduler exiting...
2021-05-21T19:18:41.226Z pid=28417 tid=kkd INFO: Got TERM, shutting down process...
2021-05-21T19:18:41.331Z pid=28417 tid=kkd INFO: Pausing to allow workers to finish...
2021-05-21T19:19:00.195Z pid=28417 tid=kot class=HardWorker jid=36a961b48d008ed18e6098f5 INFO: HardWorker: Finished
2021-05-21T19:19:00.195Z pid=28417 tid=kot class=HardWorker jid=36a961b48d008ed18e6098f5 elapsed=20.005 INFO: done

It scheduled only one job, the last one still saved in Redis

add_job.rb

require 'securerandom'
require 'json'
require 'redis'

duration = 10
duration = ARGV[0].to_i if ARGV.size > 0

redis = Redis.new(:url => 'redis://localhost')
msg = { "class" => 'HardWorker',
        "queue" => 'default',
        "args" => [duration],
        'retry' => true,
        'jid' => SecureRandom.hex(12),
        'created_at' => Time.now.to_f,
        'enqueued_at' => Time.now.to_f }
redis.lpush("queue:default", JSON.dump(msg))

deployment.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: sidekiq
spec:
  replicas: 3
  selector:
    matchLabels:
      name: sidekiq
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0%
  template:
    metadata:
      labels:
        name: sidekiq
    spec:
      terminationGracePeriod: 45 # One and a half of the Sidekiq timeout
      containers:
        - name: sidekiq
          image: <docker image>
          args:
            - bundle
            - exec
            - sidekiq
            - -r ./main.rb
            - -t
            - 30
            - -c
            - 8
          resources:
            requests:
              cpu: 200m
              memory: 1Gi
            limits:
              cpu: 1000m
              memory: 1Gi
          startupProbe:  # Will work in Kubernetes clusters from v1.20
            exec:
              command:
                - cat
                - /app/tmp/sidekiq.run
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            exec:
              command:
                - cat
                - /app/tmp/sidekiq.run
            initialDelaySeconds: 10
            periodSeconds: 30
            failureThreshold: 1
            successThreshold: 1

Gemfile

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }

gem "sidekiq"

hard_worker.rb

class HardWorker
  include Sidekiq::Worker

  def perform(timeout=10)
    Sidekiq.logger.info "HardWorker: sleep for #{timeout}"
    sleep timeout.to_i
    Sidekiq.logger.info "HardWorker: Finished"
  end
end

main.rb

require "sidekiq"
require "fileutils"

def on_startup
  FileUtils.touch("tmp/sidekiq.started")
end

Sidekiq.configure_server do |config|
  # runs after your app has finished initializing but before any jobs are dispatched.
  config.on(:startup) do
    Sidekiq.logger.info "Ready for processing"
  end

  config.on(:quiet) do
    Sidekiq.logger.info "Got TSTP, stopping further job processing..."
  end

  config.on(:shutdown) do
    Sidekiq.logger.info "Got TERM, shutting down process..."
  end

  # Cleanup all jobs
  # Sidekiq.logger.info "> Cleaing current JOBS from queues"
  # Sidekiq::Queue.new.clear
end

Sidekiq.default_worker_options = { 'retry' => false }

require_relative "worker/hard_worker"
# require_relative "worker/could_not_load_class"