TPL resque QuickStart

resque-quickstart.md

resque QuickStart

Overview

1. Install resque & Redis
2. Config resque
3. Add rake tasks
4. Implement
5. Deploy
6. Monitor

Installation

Install redis, and gem resque

gem 'resque'

Config resque

Add file app/config/initializers/resque.rb

# load resque job files  
Dir[File.join(Rails.root, 'app', 'jobs', '*.rb')].each { |file| require file }
	
# load redis connection config
config = YAML::load(File.open("#{Rails.root}/config/redis.yml"))[Rails.env]
	
# initialize resque
Resque.redis = Redis.new(:host => config['host'], :port => config['port'])

Add file config/redis.yml

default: &DEFAULTS
  host: localhost
  port: 6379

development:
  <<: *DEFAULTS

production:
  <<: *DEFAULTS

test:
  <<: *DEFAULTS
  

Add rake tasks

Add file lib/tasks/resque.rb

require 'resque/tasks'
# Start a worker with proper env vars and output redirection
def run_worker(queue, count = 5)
  puts "Starting #{count} worker(s) with QUEUE: #{queue}"
  ops = {:pgroup => true, :err => [(Rails.root + "log/workers_error.log").to_s, "a"],
         :out => [(Rails.root + "log/workers.log").to_s, "a"]}
         env_vars = {"QUEUE" => queue.to_s}
         count.times {
           ## Using Kernel.spawn and Process.detach because regular system() call would
           ## cause the processes to quit when capistrano finishes
           pid = spawn(env_vars, "rake resque:work", ops)
           Process.detach(pid)
         }
end

namespace :resque do
  task :setup => :environment do
    require File.join(File.dirname(__FILE__), '..', '..', 'config', 'environment')
    Rails.logger.info("event=resque_setup_rails_env=#{Rails.env}")
    Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }
  end

  desc "Restart running workers"
  task :restart_workers => :environment do
    Rake::Task['resque:stop_workers'].invoke
    Rake::Task['resque:start_workers'].invoke
  end

  desc "Quit running workers"
  task :stop_workers => :environment do
    pids = Array.new
    Resque.workers.each do |worker|
      pids.concat(worker.worker_pids)
    end
    if pids.empty?
      puts "No workers to kill"
    else
      syscmd = "kill -s QUIT #{pids.join(' ')}"
      puts "Running syscmd: #{syscmd}"
      system(syscmd)
    end
  end

  desc "Start workers"
  task :start_workers => :environment do
    max = ENV['MAX'] ? ENV['MAX'].to_i : 5 # default is 5 workers
    queue = ENV['QUEUE'] || '*' # '*' means for every queue

    run_worker(queue, max)
  end
end

Implement

Suppose we have a model Post post.rb, and we don't want to create a new record right after every put request but saving back to the queue to do it later.

class Post < ActiveRecord::Base
  attr_accessible :body, :title
end

Create a job file app/jobs/insert_post_job.rb

class InsertPostJob
  # queue name 
  @queue = :insert_post

  # this must be class method, params
  # do anything that you want in this method with provided params
  def self.perform(title, body)
    post = Post.new(:title => title, :body => body)
    post.save!
  end
end

Enqueue a job

 # just pass types that can be encoded as JSON
Resque.enqueue(InsertPostJob, params[:post][:title], params[:post][:body])

It actually puts the job into 'insert_post' queue, then later runs InsertPostJob.perform(title, body).

Deploy

Start the resque process

$ rake resque:start_workers # run with default 5 workers, for all queues
$ rake resque:start_workers MAX=10 # run with 10 workers, for all queues
$ rake resque:start_workers MAX=10 QUEUE=insert_post #run with 10 workers, only for insert_post job

Monitor

Resque provide a web UI to monitor all jobs in queue and their progress

$ resque-web # connect to local redis server 
$ resque-web -r redis://@localhost:6379 # connect using custom redis connection string
$ resque-web -K # kill monitor web process

For more detail https://github.com/defunkt/resque