Remove duplicate jobs from a Resque queue.

remove_duplicates.rb

queues = %w{cache contents courses export_file mailer manager notifications ranking report statused subscriptions users}
queues.collect do |queue|
  Thread.new do
    key = "queue:#{queue}"
    if Resque.redis.exists(key)
      members = Resque.redis.lrange(key, 0, -1)
      duplicate_members = members.group_by { |e| e }.select { |k, v| v.size > 1 }
      puts "#{key}: has #{duplicate_members.count} duplicates"

      if duplicate_members.count > 0
        duplicate_members.each do |duplicate_member|
          print '.'
          (duplicate_member.last.count - 1).times do
            Resque.redis.lrem(key, -1, duplicate_member.last.first)
          end
        end
        puts ''
      end
    end
  end
end.map(&:join)

remove_failed_duplicates.rb

key = 'failed'
members = Resque.redis.lrange(key, 0, -1); nil
duplicate_members = members.group_by { |e| JSON.load(e)['payload'].to_json }.select { |k, v| v.size > 1 }; nil
puts "#{key}: has #{duplicate_members.count} duplicates"

if duplicate_members.count > 0
  threads = []
  duplicate_members.to_a.in_groups_of(16000).each do |group_members|
    threads << Thread.new do
      group_members.each do |duplicate_member|
        print '.'
        (duplicate_member.last.count - 1).times do
          Resque.redis.lrem(key, -1, duplicate_member.last.first)
        end
      end
    end
  end
  threads.map(&:join)
  puts ""
end

remove_workers.rb

Resque.workers.each(&:prune_dead_workers).map(&:unregister_worker)

RESQUE_REMOVE_DUPLICATES.md

remove_duplicates.rb

Remove duplicate Resque jobs that have already been queued for processing.

Execute

To run this against a Rails installtion in production use the following command: bundle exec rails runner -e production /path/to/script/remove_duplicates.rb. See rails runner for more information.

Notes

Ideally no workers should be running while this script is running. This is NOT meant to be a real-time solution. That is, this script is not meant to be run continuously to fix an issue where a queue constantly injects duplicates into the queue. If that is the problem you are encountering please see the resque-loner gem. This is meant to be a 1 time solution for a situation where a large queue was manually created and would be difficult to recreate again without the duplicates.

Caveats

This is not a great method for removing duplicates from a queue; and could possibly take a significant amount of time on a queue with >1M records. But it will eventually de-dup the queue. See: Removing Duplicates from a List.