Skip to content

Instantly share code, notes, and snippets.

@rcanand
Created September 24, 2010 02:49
Show Gist options
  • Save rcanand/594782 to your computer and use it in GitHub Desktop.
Save rcanand/594782 to your computer and use it in GitHub Desktop.
## /config/initializers/dynamic_job.rb
require 'heroku'
# base class for all jobs that you wish to automatically scale and go down in Heroku
class DynamicJob
#set a cap on maximum number of users ever - just in case.
MAX_CONCURRENT_WORKERS = 100
def initialize
#assumption: the job class is new-ed up before enqueueing, which is always the case
workers_needed = [Delayed::Job.count + 1, MAX_CONCURRENT_WORKERS].min
if(ENV["RAILS_ENV"] == "production")
username = ENV['HEROKU_USERNAME']
password = ENV['HEROKU_PASSWORD']
app = ENV['HEROKU_APP']
client = Heroku::Client.new(username, password)
client.set_workers(app, workers_needed)
end
end
# call this from the derived class when you are done.
def clean_up
#assumption: this method is always called from within a derived job class
workers_needed = [Delayed::Job.count - 1, 0].max
if(ENV["RAILS_ENV"] == "production")
username = ENV['HEROKU_USERNAME']
password = ENV['HEROKU_PASSWORD']
app = ENV['HEROKU_APP']
client = Heroku::Client.new(username, password)
client.set_workers(app, workers_needed)
end
end
end
#example job class LoadImagesJob
class LoadImagesJob < DynamicJob
def initialize topic_id
@topic = Topic.find(topic_id)
#remember to call super() with parentheses to call superclass initializer without parameters
super()
end
def perform
@topic.load_images!
clean_up
end
end
#example enqueue
job = LoadImageJob.new topic.id # => calls some.do_something later
Delayed::Job.enqueue(job)
@timirwin
Copy link

Um, it looks like you are not decrementing correctly. If I had 500 jobs, initially I'd have only 100 workers, but it'd increase it to 499 after the first one completed. Also, do you know if Heroku is smart about which workers are terminated when decreasing the count? Or, will it pick workers at random, which may be in the middle of processing?

@rcanand
Copy link
Author

rcanand commented Oct 26, 2010

Yes, the value must be configured to be larger than anything you expect to see in production. The config is provided only for the off chance that a bug causes workers to never be decremented and we want to limit the number of workers so we dont lose our savings:)
Afaik, heroku will try to release workers gracefully, and will then send a force abort after some time. I dont remember the details, but I think an unfinished job whose worker is killed gets picked up by a free worker anyway, so it doesnt matter if an active worker is dropped. Lot of this is based on my testing a few weeks ago, so it may change.if heroku changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment