Patch to unicorn example init script: better upgrade task
From 233f504a39aca6e8b2d9332ef324b4c5cc397eca Mon Sep 17 00:00:00 2001 | |
From: Aaron Suggs <aaron@ktheory.com> | |
Date: Fri, 6 Apr 2012 18:35:38 -0400 | |
Subject: [PATCH 1/2] Whitespace: tabs to spaces | |
--- | |
examples/init.sh | 90 +++++++++++++++++++++++++++--------------------------- | |
1 files changed, 45 insertions(+), 45 deletions(-) | |
diff --git a/examples/init.sh b/examples/init.sh | |
index 1f0e035..e8f768a 100644 | |
--- a/examples/init.sh | |
+++ b/examples/init.sh | |
@@ -19,56 +19,56 @@ old_pid="$PID.oldbin" | |
cd $APP_ROOT || exit 1 | |
sig () { | |
- test -s "$PID" && kill -$1 `cat $PID` | |
+ test -s "$PID" && kill -$1 `cat $PID` | |
} | |
oldsig () { | |
- test -s $old_pid && kill -$1 `cat $old_pid` | |
+ test -s $old_pid && kill -$1 `cat $old_pid` | |
} | |
case $action in | |
-start) | |
- sig 0 && echo >&2 "Already running" && exit 0 | |
- $CMD | |
- ;; | |
-stop) | |
- sig QUIT && exit 0 | |
- echo >&2 "Not running" | |
- ;; | |
-force-stop) | |
- sig TERM && exit 0 | |
- echo >&2 "Not running" | |
- ;; | |
-restart|reload) | |
- sig HUP && echo reloaded OK && exit 0 | |
- echo >&2 "Couldn't reload, starting '$CMD' instead" | |
- $CMD | |
- ;; | |
-upgrade) | |
- if sig USR2 && sleep 2 && sig 0 && oldsig QUIT | |
- then | |
- n=$TIMEOUT | |
- while test -s $old_pid && test $n -ge 0 | |
- do | |
- printf '.' && sleep 1 && n=$(( $n - 1 )) | |
- done | |
- echo | |
+ start) | |
+ sig 0 && echo >&2 "Already running" && exit 0 | |
+ $CMD | |
+ ;; | |
+ stop) | |
+ sig QUIT && exit 0 | |
+ echo >&2 "Not running" | |
+ ;; | |
+ force-stop) | |
+ sig TERM && exit 0 | |
+ echo >&2 "Not running" | |
+ ;; | |
+ restart|reload) | |
+ sig HUP && echo reloaded OK && exit 0 | |
+ echo >&2 "Couldn't reload, starting '$CMD' instead" | |
+ $CMD | |
+ ;; | |
+ upgrade) | |
+ if sig USR2 && sleep 2 && sig 0 && oldsig QUIT | |
+ then | |
+ n=$TIMEOUT | |
+ while test -s $old_pid && test $n -ge 0 | |
+ do | |
+ printf '.' && sleep 1 && n=$(( $n - 1 )) | |
+ done | |
+ echo | |
- if test $n -lt 0 && test -s $old_pid | |
- then | |
- echo >&2 "$old_pid still exists after $TIMEOUT seconds" | |
- exit 1 | |
- fi | |
- exit 0 | |
- fi | |
- echo >&2 "Couldn't upgrade, starting '$CMD' instead" | |
- $CMD | |
- ;; | |
-reopen-logs) | |
- sig USR1 | |
- ;; | |
-*) | |
- echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>" | |
- exit 1 | |
- ;; | |
+ if test $n -lt 0 && test -s $old_pid | |
+ then | |
+ echo >&2 "$old_pid still exists after $TIMEOUT seconds" | |
+ exit 1 | |
+ fi | |
+ exit 0 | |
+ fi | |
+ echo >&2 "Couldn't upgrade, starting '$CMD' instead" | |
+ $CMD | |
+ ;; | |
+ reopen-logs) | |
+ sig USR1 | |
+ ;; | |
+ *) | |
+ echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>" | |
+ exit 1 | |
+ ;; | |
esac | |
-- | |
1.7.4.4 |
From d4d34eff222dad71c150f0e990a474af143fa3e7 Mon Sep 17 00:00:00 2001 | |
From: Aaron Suggs <aaron@ktheory.com> | |
Date: Fri, 6 Apr 2012 18:41:47 -0400 | |
Subject: [PATCH 2/2] Better upgrade script in example init script | |
Removes the sleep while waiting for the new pid file to be written. | |
(This could be problematic on busy or small servers) | |
When preload_app is enabled, reduces the time requests hang while | |
waiting for the new workers. | |
--- | |
examples/init.sh | 16 +++++++++++++++- | |
1 files changed, 15 insertions(+), 1 deletions(-) | |
diff --git a/examples/init.sh b/examples/init.sh | |
index e8f768a..3d1ba01 100644 | |
--- a/examples/init.sh | |
+++ b/examples/init.sh | |
@@ -45,8 +45,22 @@ case $action in | |
$CMD | |
;; | |
upgrade) | |
- if sig USR2 && sleep 2 && sig 0 && oldsig QUIT | |
+ if sig USR2 | |
then | |
+ printf 'Waiting for new workers' | |
+ n=$TIMEOUT | |
+ while (! (test -s $PID && ps --no-headers --ppid `cat $PID` > /dev/null)) && test $n -ge 0 | |
+ do | |
+ printf '.' && sleep 1 && n=$(( $n - 1 )) | |
+ done | |
+ if test ! -s $old_pid | |
+ then | |
+ echo | |
+ echo >&2 'New workers failed to start; see error log' | |
+ exit 1 | |
+ fi | |
+ | |
+ printf '\nStopping old master' && oldsig QUIT | |
n=$TIMEOUT | |
while test -s $old_pid && test $n -ge 0 | |
do | |
-- | |
1.7.4.4 |
I've been using the example init.sh script distributed with unicorn. | |
It works very well, but I ran in to some rough edges with the | |
`upgrade` task (which my organization runs several times per day when | |
deploying new code). | |
The salient part of the current upgrade task is: | |
sig USR2 && sleep 2 && sig 0 && oldsig QUIT | |
# then wait for the old pid to disappear | |
We found that on tiny or busy servers, the "sleep 2" was too short. A | |
too short sleep leaves the server in an undesirable state. The init | |
script fails, and both the old and new unicorn processes are running. | |
To resolve, I'd manually send a QUIT to the old master to clean things | |
up. | |
We tried increasing the sleep time, and this issue became less common, | |
but still happens. I'd like to avoid the sleep altogether. | |
Second, the old workers are terminated before the new workers are | |
ready to handle requests. This causes requests to hang (for about 25 | |
seconds in our case) until the app is loaded and the new workers begin | |
responding to requests. I'd like to minimize the impact that code | |
deploys have on our app's performance. | |
With this patch, the `upgrade` task: | |
1. Sends a USR2 signal to the current (old) master. This seems to | |
immediately rename the pid to pid.oldbin. | |
2. Waits for the master pid to exist, and for that process to have | |
children. When preload_app is true, the presence of child processes | |
means that those workers are nearly ready to handle requests (afaik, | |
they just need to execute the after_fork block) | |
3. Once the master process has children, send a QUIT to the old | |
master. (If the old master is already gone, that means the new master | |
failed to start. Exit with an error. Otherwise, wait for the old | |
master to spin down.) | |
Caveats: with my patch, it's more likely that for a second both old | |
any new workers are responding to requests. We find that this doesn't | |
usually happen, and it's not bad if it does. The way I find child | |
processes "ps --no-headers --ppid `cat $PID`" is a totally linux-ism. | |
If someone can suggest a more portable command (OS X, etc), I'd | |
appreciate it. | |
I also published the patches on github, if that's your thing: | |
https://github.com/ktheory/unicorn/tree/better_init | |
(The first patch converts tabs to spaces, as is common for ruby projects). | |
I realize this patch may be particular to our use case and may not | |
generally appropriate. I'd appreciate learning what scripts others use | |
for fast, reliable unicorn upgrades (capistrano-unicorn gem looks | |
neat). | |
Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment