gistfile1.md

We built sched.do for Yammer as an example open-sourced Rails application using the Yammer API. Like many services, Yammer API calls are subject to rate-limitations.

At times, the application would hit the Yammer API rate limit and encounter a 429 status code. To solve this, we leveraged delayed_job's functionality of retrying failed jobs. Our goal was to set the retry time to just outside the rate-limit time window.

To accomplish this, we overrode the reschedule_at and max_attempts methods, which delayed_job exposes for custom jobs. We also wanted to retry the job until success, so max_attempts was also overridden.

Code example:

module YammerRateLimited
  RATE_LIMIT_RETRIES = 50

  def error(job, exception)
    if ExceptionSilencer.is_rate_limit?(exception)
      @rate_limited = true
    else
      @rate_limited = false
      Airbrake.notify(exception)
    end
  end

  def failure(job)
    Airbrake.notify(error_message: "Job failure: #{job.last_error}")
  end

  def reschedule_at(attempts, time)
    if @rate_limited
      30.seconds.from_now
    end
  end

  def max_attempts
    if @rate_limited
      RATE_LIMIT_RETRIES
    else
      Delayed::Worker.max_attempts
    end
  end
end

Then we include this module into our custom job, like so:

class ReminderCreatedJob < Struct.new(:reminder_id)
  include YammerRateLimited

  def self.enqueue(reminder)
    job = new(reminder.id)

    Delayed::Job.enqueue(job)
  end

  def perform
    Reminder.find(reminder_id).deliver
  end
end

When a job fails due to rate-limitations, we retry it every thirty seconds, fifty times in a row until it works. The job can still fail after 50 swings, at which point we will get notified via Airbrake to take a look manually.