Tutum Zero Downtime Re-deploy
I tweeted Tutum last night asking if they're looking at implementing zero downtime re-deploys for a given service. Slightly surprised by their response as it seems like a critical feature if you want to use the service for a production environment.
"not a top priority, but by Spring :)"
As Tutum currently doesn't support graceful termination of containers within a service, I was experiencing a 5-10 second window of 503
errors, so decided to use the following hack (code below) until the feature is officially implemented.
- Create two identical web app services.
- Link HAProxy service to both.
- Redeploy one service successfully.
- Wait 10 seconds.
- Redeploy the second service.
Note - use Tutum's HAProxy image so that it re-configures to new containers, and set the POLLING_PERIOD
environment variable to 1
.
- Could instead have a single service and redeploy the individual containers.
- The code could be better, would be better in a Thor app.
- Tutum is awesome!
We've changed how redeploy works in Tutum to minimize downtime.... containers are now redeployed sequentially (a container doesn't go to terminated until another one is starting). I tried it using this script
with a load-balanced service running 5 containers, and only had 2 requests experience downtime. Not a perfect solution, but a step closer to 0-downtime. Hope that helps!