Resque + god + capistrano recipe

readme.md

Tonight littl' recipe: configuring a resque worker to be monitored via god

Here are the steps roughly:

• install god (sudo gem install god)
• sudo mkdir /etc/god then put the modified master.conf into /etc/god and try god out to see if everything works fine
• make god reboot ready
  • sudo cp god-service /etc/init.d
  • sudo chmod +x /etc/init.d/god-service
  • sudo update-rc god-service defaults
• try out your init.d script by calling /etc/init.d/god-service start|stop|status
• prepare your app resque.god configuration
• add the god part to your config/deploy.rb

You may want to tweak the deploy.rb to match more closely the expected behaviour.

Also (as suggested earlier on the resque librelist) you can tweak the configuration using ERB to create n workers instead of one here.

cheers,

-- Thibaut

deploy.rb

# note - you may need to split into a before-deploy (stop) and after-deploy (start) depending on your setup

desc "Hot-reload God configuration for the Resque worker"
deploy.task :reload_god_config do
  sudo "god stop resque"
  sudo "god load #{File.join deploy_to, 'current', 'config', 'resque.god'}"
  sudo "god start resque"
end

after 'deploy:update_code', 'deploy:update_shared_symlinks'
after 'deploy:update_code', :install_gems
after :deploy, 'deploy:reload_god_config'

god-service

#!/bin/bash
#
# god       Startup script for god (http://god.rubyforge.org)
#
# chkconfig: - 85 15
# description: God is an easy to configure, easy to extend monitoring \
#              framework written in Ruby.
#

CONF_DIR=/etc/god
GOD_BIN=/usr/local/bin/god
RUBY_BIN=/usr/local/bin/ruby
RETVAL=0

# Go no further if config directory is missing.
[ -d "$CONF_DIR" ] || exit 0

case "$1" in
    start)
      # Create pid directory
      $RUBY_BIN $GOD_BIN -c $CONF_DIR/master.conf
      RETVAL=$?
  ;;
    stop)
      $RUBY_BIN $GOD_BIN terminate
      RETVAL=$?
  ;;
    restart)
      $RUBY_BIN $GOD_BIN terminate
      $RUBY_BIN $GOD_BIN -c $CONF_DIR/master.conf
      RETVAL=$?
  ;;
    status)
      $RUBY_BIN $GOD_BIN status
      RETVAL=$?
  ;;
    *)
      echo "Usage: god {start|stop|restart|status}"
      exit 1
  ;;
esac

exit $RETVAL

master.conf

load "/var/sites/my-app/current/config/resque.god"

resque.god

God.watch do |w|
  w.name          = 'resque'
  w.interval      = 30.seconds
  w.env           = { 'RAILS_ENV' => 'production',
                      'QUEUE'     => '*' }
  w.uid           = 'user'
  w.gid           = 'user'
  w.dir           = File.expand_path(File.join(File.dirname(__FILE__),'..'))
  w.start         = "/usr/local/bin/rake resque:work"
  w.start_grace   = 10.seconds
  w.log           = File.expand_path(File.join(File.dirname(__FILE__), '..','log','resque-worker.log'))
  
  # restart if memory gets too high
  w.transition(:up, :restart) do |on|
    on.condition(:memory_usage) do |c|
      c.above = 200.megabytes
      c.times = 2
    end
  end

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
      c.interval = 5.seconds
    end

    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
      c.interval = 5.seconds
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_running) do |c|
      c.running = false
    end
  end
end