Patch to unicorn example init script: better upgrade task

0001-Whitespace-tabs-to-spaces.patch

From 233f504a39aca6e8b2d9332ef324b4c5cc397eca Mon Sep 17 00:00:00 2001
From: Aaron Suggs <aaron@ktheory.com>
Date: Fri, 6 Apr 2012 18:35:38 -0400
Subject: [PATCH 1/2] Whitespace: tabs to spaces

---
 examples/init.sh |   90 +++++++++++++++++++++++++++---------------------------
 1 files changed, 45 insertions(+), 45 deletions(-)

diff --git a/examples/init.sh b/examples/init.sh
index 1f0e035..e8f768a 100644
--- a/examples/init.sh
+++ b/examples/init.sh
@@ -19,56 +19,56 @@ old_pid="$PID.oldbin"
 cd $APP_ROOT || exit 1
 
 sig () {
-	test -s "$PID" && kill -$1 `cat $PID`
+  test -s "$PID" && kill -$1 `cat $PID`
 }
 
 oldsig () {
-	test -s $old_pid && kill -$1 `cat $old_pid`
+  test -s $old_pid && kill -$1 `cat $old_pid`
 }
 
 case $action in
-start)
-	sig 0 && echo >&2 "Already running" && exit 0
-	$CMD
-	;;
-stop)
-	sig QUIT && exit 0
-	echo >&2 "Not running"
-	;;
-force-stop)
-	sig TERM && exit 0
-	echo >&2 "Not running"
-	;;
-restart|reload)
-	sig HUP && echo reloaded OK && exit 0
-	echo >&2 "Couldn't reload, starting '$CMD' instead"
-	$CMD
-	;;
-upgrade)
-	if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
-	then
-		n=$TIMEOUT
-		while test -s $old_pid && test $n -ge 0
-		do
-			printf '.' && sleep 1 && n=$(( $n - 1 ))
-		done
-		echo
+  start)
+    sig 0 && echo >&2 "Already running" && exit 0
+    $CMD
+    ;;
+  stop)
+    sig QUIT && exit 0
+    echo >&2 "Not running"
+    ;;
+  force-stop)
+    sig TERM && exit 0
+    echo >&2 "Not running"
+    ;;
+  restart|reload)
+    sig HUP && echo reloaded OK && exit 0
+    echo >&2 "Couldn't reload, starting '$CMD' instead"
+    $CMD
+    ;;
+  upgrade)
+    if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
+    then
+      n=$TIMEOUT
+      while test -s $old_pid && test $n -ge 0
+      do
+        printf '.' && sleep 1 && n=$(( $n - 1 ))
+      done
+      echo
 
-		if test $n -lt 0 && test -s $old_pid
-		then
-			echo >&2 "$old_pid still exists after $TIMEOUT seconds"
-			exit 1
-		fi
-		exit 0
-	fi
-	echo >&2 "Couldn't upgrade, starting '$CMD' instead"
-	$CMD
-	;;
-reopen-logs)
-	sig USR1
-	;;
-*)
-	echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
-	exit 1
-	;;
+      if test $n -lt 0 && test -s $old_pid
+      then
+        echo >&2 "$old_pid still exists after $TIMEOUT seconds"
+        exit 1
+      fi
+      exit 0
+    fi
+    echo >&2 "Couldn't upgrade, starting '$CMD' instead"
+    $CMD
+    ;;
+  reopen-logs)
+    sig USR1
+    ;;
+  *)
+    echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
+    exit 1
+    ;;
 esac
-- 
1.7.4.4

0002-Better-upgrade-script-in-example-init-script.patch

From d4d34eff222dad71c150f0e990a474af143fa3e7 Mon Sep 17 00:00:00 2001
From: Aaron Suggs <aaron@ktheory.com>
Date: Fri, 6 Apr 2012 18:41:47 -0400
Subject: [PATCH 2/2] Better upgrade script in example init script

Removes the sleep while waiting for the new pid file to be written.
(This could be problematic on busy or small servers)

When preload_app is enabled, reduces the time requests hang while
waiting for the new workers.
---
 examples/init.sh |   16 +++++++++++++++-
 1 files changed, 15 insertions(+), 1 deletions(-)

diff --git a/examples/init.sh b/examples/init.sh
index e8f768a..3d1ba01 100644
--- a/examples/init.sh
+++ b/examples/init.sh
@@ -45,8 +45,22 @@ case $action in
     $CMD
     ;;
   upgrade)
-    if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
+    if sig USR2
     then
+      printf 'Waiting for new workers'
+      n=$TIMEOUT
+      while (! (test -s $PID && ps --no-headers --ppid `cat $PID` > /dev/null)) && test $n -ge 0
+      do
+        printf '.' && sleep 1 && n=$(( $n - 1 ))
+      done
+      if test ! -s $old_pid
+      then
+        echo
+        echo >&2 'New workers failed to start; see error log'
+        exit 1
+      fi
+
+      printf '\nStopping old master' && oldsig QUIT
       n=$TIMEOUT
       while test -s $old_pid && test $n -ge 0
       do
-- 
1.7.4.4

README.txt

I've been using the example init.sh script distributed with unicorn.
It works very well, but I ran in to some rough edges with the
`upgrade` task (which my organization runs several times per day when
deploying new code).

The salient part of the current upgrade task is:

   sig USR2 && sleep 2 && sig 0 && oldsig QUIT
   # then wait for the old pid to disappear

We found that on tiny or busy servers, the "sleep 2" was too short. A
too short sleep leaves the server in an undesirable state. The init
script fails, and both the old and new unicorn processes are running.
To resolve, I'd manually send a QUIT to the old master to clean things
up.

We tried increasing the sleep time, and this issue became less common,
but still happens. I'd like to avoid the sleep altogether.

Second, the old workers are terminated before the new workers are
ready to handle requests. This causes requests to hang (for about 25
seconds in our case) until the app is loaded and the new workers begin
responding to requests. I'd like to minimize the impact that code
deploys have on our app's performance.

With this patch, the `upgrade` task:

1. Sends a USR2 signal to the current (old) master. This seems to
immediately rename the pid to pid.oldbin.
2. Waits for the master pid to exist, and for that process to have
children. When preload_app is true, the presence of child processes
means that those workers are nearly ready to handle requests (afaik,
they just need to execute the after_fork block)
3. Once the master process has children, send a QUIT to the old
master. (If the old master is already gone, that means the new master
failed to start. Exit with an error. Otherwise, wait for the old
master to spin down.)

Caveats: with my patch, it's more likely that for a second both old
any new workers are responding to requests. We find that this doesn't
usually happen, and it's not bad if it does. The way I find child
processes "ps --no-headers --ppid `cat $PID`" is a totally linux-ism.
If someone can suggest a more portable command (OS X, etc), I'd
appreciate it.

I also published the patches on github, if that's your thing:
https://github.com/ktheory/unicorn/tree/better_init
(The first patch converts tabs to spaces, as is common for ruby projects).

I realize this patch may be particular to our use case and may not
generally appropriate. I'd appreciate learning what scripts others use
for fast, reliable unicorn upgrades (capistrano-unicorn gem looks
neat).

Thanks!