Setup Monitoring for a rake ts:dd job (Thinking Sphinx Delayed Delta)

README.txt

The purpose of all this is to see if sphinx, thinking sphinx and thinking sphinx delayed delta are all working properly.

I created a test controller on a separate monit subdomain that simply generates and posts a test value and then uses curl to retrieve it. If the two values match, then sphinx is working properly with delayed delta.

This example assumes a Linux installation.

The file 'delayed_delta.sh' spawns the `rake ts:dd` process in the background, saving its PID to tmp/pids in your Rails project. You can start and stop it by running '/etc/init.d/delayed_delta.sh start' and '/etc/init.d/delayed_delta.sh stop'. You will use these in your monitoring to, see the monitrc snippet.

In a crontab, every X seconds or minutes, run 'ar_sphinx_mon.sh' to see if records are properly being inserted and indexed. If they aren't, then kill all Thinking Sphinx processes and monit should restart them.

To make a test post, just point your browser to 'http://monit.mydomain.com/testpost?testval=helloworld' to retrieve it 'http://monit.mydomain.com/test?testval=helloworld'. The latter testval is actually a search parameter.

ar_sphinx_mon.sh

#! /bin/sh
POST_URL="http://monit.mydomain.com/testpost?testval="
GET_URL="http://monit.mydomain.com/test?testval="

timestamp=`date +%s`

POST_URL=$POST_URL$timestamp

code=$(curl $POST_URL)

#counter=0
#if [ $counter -lt 3 ]

	if [ $? -ne 0 ]
	then
		echo "POST: $POST_URL: [FAIL]"
		exit 1
	fi

	sleep 10

	GET_URL=$GET_URL$timestamp

	response=$(curl $GET_URL)

	if [ $? -eq 0 ]
	then
		if [ "$response" -eq "$timestamp" ]
		then
			echo "SUCCESS!"
			exit 0
		else
			echo "GET: $GET_URL: [STRING MISMATCH]"
			exit 3
		fi
	else
		echo "GET: $GET_URL: [FAIL]"
		exit 2
	fi

#	let "counter++"
#	sleep 30
#fi

# kill rake ts:dd
/etc/init.d/delayed_delta stop

# kill sphinx
su - deploy -c "cd /home/deploy/rails_root && rake RAILS_ENV=production ts:stop"

# monit should restart them

exit 4

delayed_delta.sh

#! /bin/sh
RAILS_ROOT="/home/deploy/rails_root"
RAKE="rake"
ENV="production"
PID_FILE="$RAILS_ROOT/tmp/pids/rake_ts_dd.pid"
LOG_FILE="$RAILS_ROOT/log/rake_ts_dd.log"

case "$1" in
    start)
    echo "Starting rake ts:dd: "

    if [ -e $PID_FILE ]
    then
        pid=`cat $PID_FILE`
        ps -ef | awk -v pid=$pid 'pid == $2 {print $0}' | grep 'rake ts:dd' > /dev/null
        if [ $? -eq 0 ]
        then
            echo "It appears that rake ts:dd is already running on PID $pid" #>> $LOG_FILE 2>&1
            exit 1
        else
            echo "Removing Stale PID File" #>> $LOG_FILE 2>&1
            rm $PID_FILE
        fi
    fi

    su - deploy -c "cd $RAILS_ROOT; RAILS_ENV=$ENV $RAKE ts:dd & echo \$! > $PID_FILE"
    echo "rake ts:dd started with PID `cat $PID_FILE`!" #>> $LOG_FILE
    ;;
    stop)
    echo "Stopping rake ts:dd: "
    if [ ! -e $PID_FILE ]
    then
        echo "$PID_FILE not found."
        exit 1
    fi
    pid=`cat $PID_FILE`
    #su - deploy -c "kill -9 $pid" #>> $LOG_FILE 2>&1
    kill -9 $pid
    ret=$?
    if [ $ret -eq 0 ]
    then
        echo "Removing PID file"
        rm $PID_FILE
        echo "Successfully shutdown rake ts:dd" #>> $LOG_FILE 2>&1
        exit 0
    else
        echo "Couldn't kill process `cat $PID_FILE`"
        exit 1
    fi
    ;;
    *)
    echo "Usage: $N {start|stop}" >&2
    exit 1
    ;;
esac

exit 0

monit.rb

class Monit < ActiveRecord::Base
	define_index do
	  indexes testval
	  set_property :delta => :delayed
	end
end

monit_controller.rb

class MonitController < ApplicationController
	layout false

	def index
 		@monit = Monit.search("#{params[:testval]}").first
		render :text => @monit.try(:testval)
	end

	def post_test
		Monit.destroy_all
		@monit = Monit.create(:testval => params[:testval])
		if @monit
			render :text => 0	# stick with bash success codes
		else
			render :text => 1	# stick with bash error codes
		end
	end
end

monit_migration.rb

class CreateMonits < ActiveRecord::Migration
  def self.up
    create_table :monits do |t|
      t.string :testval
      t.boolean :delta, :default => true, :null => false
      t.timestamps
    end
  end

  def self.down
    drop_table :monits
  end
end

monitrc

check process ts_dd with pidfile /home/deploy/Flash/tmp/pids/rake_ts_dd.pid
    stop program = "/etc/init.d/delayed_delta stop"
    start program = "/etc/init.d/delayed_delta start"

routes.rb

  ### ------------ Begin monit subdomain route directives ------------ ###
  constraints(:subdomain => /monit/) do
    match '/' => 'monit#index', :as => :monit_root
    get '/test' => 'monit#index', :as => :monit_index
    match '/testpost' => 'monit#post_test', :as => :monit_testpost
    match '(/*everything_else)' => 'monit#index'
  end # Monit subdomain