monitor-redis-resque.rb

# Redis
%w{6379}.each do |port|
  God.watch do |w|
    w.name          = "redis"
    w.interval      = 30.seconds
    w.start         = "/etc/init.d/redis start"
    w.stop          = "/etc/init.d/redis stop"
    w.restart       = "/etc/init.d/redis restart"
    w.start_grace   = 10.seconds
    w.restart_grace = 10.seconds

    w.start_if do |start|
      start.condition(:process_running) do |c|
          c.interval = 5.seconds
          c.running = false
      end
    end
  end
end

# Resque
God.watch do |w|
  w.name          = "resque-1.8.0"
  w.interval      = 30.seconds
  w.start         = "cd /var/www/apps/limitedpressing/current && rake environment RAILS_ENV=production resque:work QUEUE=high,medium,low"
  w.start_grace   = 10.seconds
  
  # retart if memory gets too high
  w.transition(:up, :restart) do |on|
    on.condition(:memory_usage) do |c|
      c.above = 350.megabytes
      c.times = 2
    end
  end
  
  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end
  
  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
      c.interval = 5.seconds
    end
    
    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
      c.interval = 5.seconds
    end
  end
  
  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_running) do |c|
      c.running = false
    end
  end
end

I don't think your redis god conf works here. Besides the fact that the port loop doesn't do anything, God needs to start the process itself or know what the process's PID is. your PID here is not redis-server's but instead the script that launches it (if I'm not mistaken). I had run this god script and my /var/run/god/redis.pid kept changing despite the fact that redis was working fine (on another pid). Anyways: for me at least, god wasn't doing anything besides continually trying to relaunch redis while it was already running.

That happened to me at first as well (god not using the correct pid and continually starting new resque workers) until I found that I had an & at the end of my w.start command. When I removed the & at the end of my w.start command, god started writing the correct process id to the pid file and prevented it from continually starting new workers.

If you kill -QUIT the process with the id in your resque pid (that god creates), god will start 1 new resque process and update the pid file. At least it's doing it properly for me with this setup. I'd definitely double check you aren't trying to start resque in the background with an & in your w.start command. I know, it's silly but that's what I missed at first.

You're right the port loop for redis doesn't do anything. I'll change that, thanks.

(sorry I posted with the wrong account earlier).

I was using your /etc/init.d/redis script -- which daemonizes redis. Just adding a block with redis's pid file fixes the problem.

w.pid_file = '/var/run/redis.pid'
w.behavior(:clean_pid_file)

thanks

Ahhh, I apologize. I thought you were talking about resque restarting. Very odd, I don't have any issue with redis continually being started by god with this setup, but thanks for the update!

no the problem isn't redis restarting. it's god continually trying to restart redis (while it's running happily). check your /var/run/god/redis.pid and see if it matches the redis-server's actual pid.

Okay, I totally misunderstood, sorry. I had a problem where god would keep starting new resque workers so I thought you were talking about that.

You're right, the pid is wrong and also my /etc/init.d/redis script is creating a pid at /var/run/redis.pid whereas god is using /var/run/god/redis.pid. Everything's all confused haha.

So I guess the God.watch can be moved out the of the useless port loop, like you mentioned earlier. Then those two lines you added above: w.pid_file and w.behavior, can those be added within the God.watch block?

checkout my fork, it has my changes. but the answer is: yes :)

p.s. thanks for the blogpost regardless. was useful for getting started. truth be told, I think god docs are lacking.

Awesome. Thanks for the fork. I'll pull in your changes and update my post. And no problem, I mostly had a problem figuring out how to get the resque workers to get restarted on deploy though. That was a real pain in the ass for me. Thanks again!