Recovering from Laravel failed-job Hell

blog.md

Update: After a PR, this has been pulled into Laravel 5.1. Now retry multiple jobs with queue:retry 1 2 3 or queue:retry all. See the new retry command.

Recently, the Amazon SES region we use to send email went down for approx 45 minutes. During that time, any emails that we had queued to run failed. This left us with a massive list of failed jobs which all needed to be released to the queue to run again.

The Problem

Currently the only way to accomplish this is to run the artisan queue:retry command for each and every failed job. This task would be made much easier if there were a way to release multiple jobs back onto the queue using a single command. Let’s make that happen! (or just skip to the final commands)

RTFM

To get started, take a look at the source of the current retry command over at the laravel/framework repo.

We can see that the retry command accepts a single ID, does a quick lookup, resets the retry count, and writes the job back onto the queue. What I would like to be able to do is pass an array of failed job ids that need to be released. As a bonus, it would be nice to have the option to release all failed jobs in a single pass. The signature for the command we build could look something like php artisan queue:retry-multiple 1,2,3,4.

While we could duplicate the current retry command and slightly change the logic to accomodate our purposes, I think that the better option is to use the existing retry command inside of our new command, writing a simple wrapper that will loop over our passed in ids.

Releasing Multiple Failed Jobs

To generate the new command, run php artisan make:console QueueRetryMultipleCommand. After generating the command, be sure to add App\Console\Commands\QueueRetryMultipleCommand::class to the $commands array in the App\Console\Kernel. Now that we have our new command available to us from the command line, lets change the boilerplate code.

Start by updating the $signature property.

protected $signature = 'queue:retry-multiple {ids : the ids of the failed-jobs you would like released (separate multiple ids with a comma).}';

The first part of the property defines what the command signature will be and the second part allows us to define our arguments and give them a description. I know the description is a bit verbose, but we want to make sure we communicate the formatting expectations to our user.

Next let's look at the handle method. This is the method that will be executed when the command is called. Let’s replace it with:

public function handle()
{
    collect($this->parseIds())->each(function($id) {
        $this->call('queue:retry', ['id' => $id]);
    });
}

private function parseIds()
{
    return explode(',', $this->argument('ids'));
}

First we are wrapping up all of the ids that have been passed to our command into a collection. Notice the private method parseIds. We don't have native array support for arguments in these commands, so having this named method helps to clear up our intent for the explode statement. After we collect all of the ids, we loop over each of them and use the $this->call() to call the original queue:retry artisan command, passing along our id as an argument. All of the output from the queue:retry command will be sent through to the console so there is no need for us to give the user additional feedback.

Releasing All Failed Jobs

To create a command that will release all of the failed queues, follow the same steps as above to create a new console commmand and add it to the console kernel. This time let’s name it QueueRetryAllCommand, set the $signature property to be queue:retry-all, and set a sensible $description. In the handle method, we will borrow some of the code from the original retry command to get any jobs that have failed, then we will loop over the ids and call the retry command. The handle method on our new command should look like this:

public function handle()
{
    $failed = $this->laravel['queue.failer']->all();

    if (! empty($failed)) {
        collect($failed)->each(function($value) {
            $this->call('queue:retry', ['id' => $value->id]);
        });
    } else {
        $this->error('No failed jobs.');
    }
}

That’s pretty much it. Hope this helps you out next time you find yourself facing a giant number of failed-jobs. 👍

Check out the final command files over at https://gist.github.com/JacobBennett/d705bf468ac1a7995a2a

gistlog.yml

published: true
preview: The Amazon SES region we use to send email went down for approx 45 minutes. During that time, any emails that we had queued to run failed. This left us with a massive list of failed jobs which all needed to be released to the queue to run again. Sure would be handy to be able to relase multiple jobs at once.