Skip to content

Instantly share code, notes, and snippets.

@jagthedrummer
Created April 5, 2017 19:58
Show Gist options
  • Save jagthedrummer/9b18b9c29a489357fb7ee1634bc2049d to your computer and use it in GitHub Desktop.
Save jagthedrummer/9b18b9c29a489357fb7ee1634bc2049d to your computer and use it in GitHub Desktop.
Rake task to transfer Sidekiq jobs from one redis instance to another
# This task should be run inside an environment that is already configured to connect to the redis
# instance that we're transfering AWAY FROM.
#
# The task should be handed the URL of the redis instance that we're MOVING TO.
#
# To run it and pass in the destination Redis you'd do something like this:
# rake sidekiq:transfer[redis://...]
#
# As jobs are added to the destination Redis, they're deleted from the source Redis. This
# allows the task to be restarted cleanly if it fails in the middle due to a network error
# or something like that.
#
namespace :sidekiq do
desc 'Transfer sidekiq jobs from one instance to another'
task :transfer, [:destination] => :environment do |task, args|
def transfer_queue queue, destination_client
puts "transfering queue : #{queue.name}"
queue.each do |job|
destination_client.push 'class' => job.klass, 'args' => job.args
job.delete
end
end
def transfer_set set, destination_client
set.each do |job|
destination_client.push 'class' => job.klass, 'args' => job.args, 'at' => job.at.to_i
scheduled_job.delete
end
end
destination_pool = ConnectionPool.new { Redis.new(url: args[:destination]) }
destination_client = Sidekiq::Client.new(destination_pool)
source_queues = Sidekiq::Queue.all
source_queues.each do |queue|
transfer_queue queue, destination_client
end
puts "transfering scheduled jobs"
ss = Sidekiq::ScheduledSet.new
transfer_set ss, destination_client
puts "transfering retried jobs"
rs = Sidekiq::RetrySet.new
transfer_set rs, destination_client
end
end
@diego-cowboy
Copy link

diego-cowboy commented Aug 8, 2022

Thanks a lot 🙌
This is my procedural version I've used today, which includes checks on queue size to validate the migration:

# Connect & check old redis queue
Sidekiq.configure_client { |config| config.redis = { url: ENV['THE_OLD_REDIS_URL'] } }
Sidekiq::ScheduledSet.new.size

# Init new redis client
dst_pool = ConnectionPool.new { Redis.new(url: ENV['THE_NEW_REDIS_URL']) }
dst_client = Sidekiq::Client.new(dst_pool)

# load old jobs
old_jobs = []
Sidekiq::ScheduledSet.new.each do |job|
  old_jobs << job
end

# push old jobs to new redis
old_jobs.each do |job|
  puts "Adding #{job.klass} with #{job.args} at #{job.at.to_i}"
  dst_client.push 'class' => job.klass, 'args' => job.args, 'at' => job.at.to_i
end

# check if new jobs were added
Sidekiq.configure_client { |config| config.redis = { url: ENV['THE_NEW_REDIS_URL'] } }
Sidekiq::ScheduledSet.new.size

# delete old job if needed
# Sidekiq.configure_client { |config| config.redis = { url: ENV['THE_OLD_REDIS_URL'] } }
# old_jobs.each do |job|
#   job.delete
# end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment