Why merge backburner with quebert?

sketch.md

Assessment of Quebert & Backburner to figure out how to merge the two.

Quebert

1. Build out an async proxy in place of #async_send so that TTR, delays, and priority can be specified in a cleaner way. This looks something like User.new.async(delay: 3).send(:notify_user) as opposed to the current User.new.async_send(:notify_user, :quebert => {delay: 3, pri: 1000})
2. Quebert supports many back-ends: InProcess, Sync, and Beanstalk. Sync and InProcess were created to aid with testing assertions so that Quebert.backend.queue.should have(1).items is possible. It only needs Beanstalk, so the multiple-backends could be removed for the sake of simplicity. A stub could be provided for peeps that want to assert jobs are thrown on the queue in a test env.
3. There's a controller layer between the Job and the Backend that communicates using exceptions, which is suppose to deal with multiple backends. Eww. I want to just pass a controller into the job and have it deal.

Backburner

Some stuff I'd change:

1. I'm not convinced that the Job mix-in is the best approach. A job should be a class itself. Mixin's make sense as syntatical sugar to make it easier to queue an instance of a class into a job, and from pulling the job off the queue and shoving it back into that class for processing.
2. I don't think having different queues per app is a good idea in a beanstalkd world. Resque does this because there's no concept of priorities in Redis. Since Beanstalkd lets people specify priories of jobs, its a moot point to run different numbers of workers for different queue names. Put more emphasis on priority to deal with this.

Stuff that's good:

1. Runners are much more sophisticated than Quebert (forking runner, etc)
2. The name.
3. The hooks, though I'd tweak the implementation of this a bit more.

Features that would be awesome

1. We had a mis-numbered priority in production once that brought our system down. We'd like to have "named" priorities that are stack-ranked. This is a simple DSL that looks like: Backburner.config.priorities = [:high, :medium, :custom_pri, :low], which is mapped to the int values that beanstalkd understands. When tossing a job on the queue, a named priority could be specified like User.new.async(pri: :high).blah.
2. Web based GUI admin tool. We could pull this in from a Ruby Sinatra beanstalkd admin tool with little effort. Would want to re-brand it with backburner so that it feels more put together.
3. CLI for querying/filtering jobs for performing batch operations on jobs that are in the queue. This is important for when things go bad in production and certain jobs with certain payloads may need to be buried until a patch can be pushed to prod and the jobs are re-run. Web GUI might be able to use this CLI interface.

I'm not convinced that the Job mix-in is the best approach. A job should be a class itself. Mixin's make sense as syntatical sugar to make it easier to queue an instance of a class into a job, and from pulling the job off the queue and shoving it back into that class for processing.

I understand your point, I debated this when I first created backburner. I'd be happy for any class that responds to perform to be a valid job. Personally I find that using the mixin is to my benefit though, I like having the syntactic sugar and also it allows me to easily aggregate a list of known job types (and the associated named tubes). Otherwise determining the known queues would be difficult. But that said, I like the idea of supporting the mixin still but not having it be a requirement.

I don't think having different queues per app is a good idea in a beanstalkd world. Resque does this because there's no concept of priorities in Redis. Since Beanstalkd lets people specify priories of jobs, its a moot point to run different numbers of workers for different queue names. Put more emphasis on priority to deal with this.

If I understand correctly, are you proposing to support still named queues and workers for specific queues (if specified) but by default to have all jobs in a single app use the same queue?

Build out an async proxy in place of #async_send so that TTR, delays, and priority can be specified in a cleaner way. This looks something like User.new.async(delay: 3).send(:notify_user) as opposed to the current User.new.async_send(:notify_user, :quebert => {delay: 3, pri: 1000})

Yeah I like the format User.async(delay: 100).notify_user I think it's the most natural of the syntaxes for async executions I've seen.

There's a controller layer between the Job and the Backend that communicates using exceptions. Eww. I want to just pass a controller into the job and have it deal.

Confused by this, why does a job have to be aware of a controller? Maybe I misunderstand

We had a mis-numbered priority in production once that brought our system down. We'd like to have "named" priorities that are stack-ranked. This is a simple DSL that looks like: Backburner.config.priorities = [:high, :medium, :custom_pri, :low], which is mapped to the int values that beanstalkd understands. When tossing a job on the queue, a named priority could be specified like User.new.async(pri: :high).blah.

I like this idea, kind of like an enum shorthand. You can specify hardcoded numbers of just use the presets. I added an issue for this to backburner: nesquena/backburner#23

Web based GUI admin tool. We could pull this in from a Ruby Sinatra beanstalkd admin tool with little effort. Would want to re-brand it with backburner so that it feels more put together.

Love this idea, here it is being planned out: nesquena/backburner#11

CLI for querying/filtering jobs for performing batch operations on jobs that are in the queue. This is important for when things go bad in production and certain jobs with certain payloads may need to be buried until a patch can be pushed to prod and the jobs are re-run. Web GUI might be able to use this CLI interface.

Interesting I can see the value in this although beanstalkd doesn't make this particularly easy since there's little visibility into the queue. But we have the same issue for the admin panel and the consensus from keith is that we can just rapidly reserve and release to enumerate jobs.

Links to the issues relating to your comments:

Quebert supports many back-ends: InProcess, Sync, and Beanstalk. Sync and InProcess were created to aid with testing assertions so that Quebert.backend.queue.should have(1).items is possible. It only needs Beanstalk, so the multiple-backends could be removed for the sake of simplicity. A stub could be provided for peeps that want to assert jobs are thrown on the queue in a test env.

nesquena/backburner#25

I'm not convinced that the Job mix-in is the best approach. A job should be a class itself. Mixin's make sense as syntatical sugar to make it easier to queue an instance of a class into a job, and from pulling the job off the queue and shoving it back into that class for processing.

nesquena/backburner#26

I don't think having different queues per app is a good idea in a beanstalkd world. Resque does this because there's no concept of priorities in Redis. Since Beanstalkd lets people specify priories of jobs, its a moot point to run different numbers of workers for different queue names. Put more emphasis on priority to deal with this.

nesquena/backburner#27

We had a mis-numbered priority in production once that brought our system down. We'd like to have "named" priorities that are stack-ranked. This is a simple DSL that looks like: Backburner.config.priorities = [:high, :medium, :custom_pri, :low], which is mapped to the int values that beanstalkd understands. When tossing a job on the queue, a named priority could be specified like User.new.async(pri: :high).blah.

nesquena/backburner#23

Web based GUI admin tool. We could pull this in from a Ruby Sinatra beanstalkd admin tool with little effort. Would want to re-brand it with backburner so that it feels more put together.

nesquena/backburner#11

CLI for querying/filtering jobs for performing batch operations on jobs that are in the queue. This is important for when things go bad in production and certain jobs with certain payloads may need to be buried until a patch can be pushed to prod and the jobs are re-run. Web GUI might be able to use this CLI interface.

nesquena/backburner#24