Sidekiq Dynamic Priorities

plan.md

Purpose

SDP will add the ability to use queues with a scored priority using the SortedSet functionality in Redis. This will allow projects to add jobs to a dynamic queue with a numeric priority score. Jobs will then be processed from highest priority to the lowest. If a job is added with the highest priority, it will then be the next job to get pulled from the queue.

This will transparently augment the functionality in Sidekiq and will only provide additive functionality.

Example

As an example, a dynamic priority queue exists with workers Alpha and Beta. At time(0) the queue exists but workers have not pulled any jobs yet:

 Time(0)
 
 Queue Alpha               Worker Alpha : NIL
 ---------------------     Worker Beta  : NIL
| priority | job      |
|        1 | DDDDDDDD |
|        5 | CCCCCCCC |
|       10 | BBBBBBBB |
|       20 | AAAAAAAA |
 ---------------------

At time(1) workers Alpha and Beta both pull the highest priority jobs from the queue:

 Time(1)
 
 Queue Alpha               Worker Alpha : AAAAAAAA
 ---------------------     Worker Beta  : BBBBBBBB
| priority | job      |
|          |          |
|          |          |
|        1 | DDDDDDDD |
|        5 | CCCCCCCC |
 ---------------------

At time(2) a job EEEEEEEE with priority 7 is added:

 Time(2)
 
 Queue Alpha               Worker Alpha : AAAAAAAA
 ---------------------     Worker Beta  : BBBBBBBB
| priority | job      |
|          |          |
|        1 | DDDDDDDD |
|        5 | CCCCCCCC |
|        7 | EEEEEEEE |
 ---------------------

At time(3) a job FFFFFFFF with priority 2 is added:

 Time(3)
 
 Queue Alpha               Worker Alpha : AAAAAAAA
 ---------------------     Worker Beta  : BBBBBBBB
| priority | job      |
|        1 | DDDDDDDD |
|        2 | FFFFFFFF |
|        5 | CCCCCCCC |
|        7 | EEEEEEEE |
 ---------------------

At time(4) a job BBBBBBBB is completed and worker Beta pulls job EEEEEEEE from the queue

 Time(4)
 
 Queue Alpha               Worker Alpha : AAAAAAAA
 ---------------------     Worker Beta  : EEEEEEEE
| priority | job      |
|          |          |
|        1 | DDDDDDDD |
|        2 | FFFFFFFF |
|        5 | CCCCCCCC |
 ---------------------

At time(5) a job AAAAAAAA is completed and worker Beta pulls job CCCCCCCC from the queue

 Time(5)
 
 Queue Alpha               Worker Alpha : AAAAAAAA
 ---------------------     Worker Beta  : EEEEEEEE
| priority | job      |
|          |          |
|          |          |
|        1 | DDDDDDDD |
|        2 | FFFFFFFF |
 ---------------------

Changes

Dynamic Queue Desigination

To mark a queue as dynamic, there will be an additional Redis list with key dynamic_queues which will sit next to the existing queues list. Any queue names that are in this list will be treated as a dynamic queue.