Bash script to install Django Q with Supervisor on Amazon Linux 2 (AL2 / Elastic Beanstalk) - UPD with Procfile in the comments

01_set_env.sh

#!/bin/bash

#Create a copy of the environment variable file.
sudo cp /opt/elasticbeanstalk/deployment/env /opt/elasticbeanstalk/deployment/custom_env_var

#Set permissions to the custom_env_var file so this file can be accessed by any user on the instance. You can restrict permissions as per your requirements.
sudo chmod 644 /opt/elasticbeanstalk/deployment/custom_env_var

#Remove duplicate files upon deployment.
sudo rm -f /opt/elasticbeanstalk/deployment/*.bak

#Install/enable extra packages
sudo amazon-linux-extras install epel -y

#Install supervisor
sudo yum install supervisor -y >/dev/null 2>&1
if [ $? -ne 1 ]; then   # Exit on any any error except 'nothing to do' 
    exit 0
fi

02_django_migrate.sh

#!/bin/bash

source /var/app/venv/*/bin/activate && {

# collect static
python manage.py collectstatic --noinput;
# log which migrations have already been applied
python manage.py showmigrations;
# migrate user model prior to other models if you have Custom User Model < NB
#python manage.py migrate users --noinput;
# migrate the rest
python manage.py migrate --noinput;
# create superuser
python manage.py createsu;

}

03_django_q.sh

#!/usr/bin/env bash

# Author: Valentine Solonechnyi <valentinesolo@gmail.com>
# Based on instructions by Rémi Héneault 
# https://gist.github.com/codeSamuraii/0e11ce6d585b3290b15a9ad163b9aa06
# Django Q supervisor configuration and setup for Amazon Linux 2

mkdir -p /var/log/djangoq/ /var/run/djangoq/

# Get django environment variables
# grep '^PYTHONPATH\|^PATH' no filtering of env variables
# djangoqenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '\n' ',' | sed 's/=/="/'g | sed 's/,/",/g'`
# fix from @matiszz
djangoqenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '\n' ',' | sed 's/=/="/'g | sed 's/,/",/g' | sed 's/="="/=="/'g | sed 's/""/"/'g`
djangoqenv=${djangoqenv%?}

# Create djangoq configuraiton script
djangoqconf="[program:django-q]
command=bash -c 'source /var/app/venv/*/bin/activate && python manage.py qcluster'
directory=/var/app/current
user=nobody
numprocs=1
stdout_logfile=/var/log/djangoq/worker.log
stderr_logfile=/var/log/djangoq/worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
stopasgroup=true
environment=$djangoqenv
"

# Create the djangoq supervisord conf script
echo "$djangoqconf" | sudo tee /etc/supervisord.d/djangoq.conf

# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "files = supervisord.d/*.conf" /etc/supervisord.conf
  then
  sed -i "s/*.ini/*.conf/g" /etc/supervisord.conf
fi

#Launch supervisord process
sudo supervisord -c /etc/supervisord.conf

# Reread the supervisord config
sudo supervisorctl reread

# Update supervisord in cache without restarting all services
sudo supervisorctl update

# Start/Restart djangoqd through supervisord.
sudo supervisorctl -c /etc/supervisord.conf restart django-q

04_supervisor_init.sh

#!/usr/bin/env bash

#Supervisor init config

# Add a new configuration of restart Supervisor
sudo cp /var/app/current/supervisord.sample /etc/init.d/supervisord

#Add execute authority
sudo chmod +x /etc/init.d/supervisord

#Add the configuration into system
sudo chkconfig --add supervisord

#Switch on the configuration and start
sudo chkconfig supervisord on
sudo service supervisord start

supervisord.sample

#! /bin/sh
### BEGIN INIT INFO
# Provides:          supervisord
# Required-Start:    $remote_fs
# Required-Stop:     $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Example initscript
# Description:       This file should be used to construct scripts to be
#                    placed in /etc/init.d.
### END INIT INFO

# Author: Dan MacKinlay <danielm@phm.gov.au>
# Based on instructions by Bertrand Mathieu
# http://zebert.blogspot.com/2009/05/installing-django-solr-varnish-and.html

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Description of the service"
NAME=supervisord
DAEMON=/usr/local/bin/supervisord
DAEMON_ARGS=""
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
	# Return
	#   0 if daemon has been started
	#   1 if daemon was already running
	#   2 if daemon could not be started
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
		|| return 1
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
		$DAEMON_ARGS \
		|| return 2
	# Add code here, if necessary, that waits for the process to be ready
	# to handle requests from services started subsequently which depend
	# on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
    # Return
	#   0 if daemon has been stopped
	#   1 if daemon was already stopped
	#   2 if daemon could not be stopped
	#   other if a failure occurred
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
	RETVAL="$?"
	[ "$RETVAL" = 2 ] && return 2
	# Wait for children to finish too if this is a daemon that forks
	# and if the daemon is only ever run from this initscript.
	# If the above conditions are not satisfied then add some other code
	# that waits for the process to drop all resources that could be
	# needed by services started subsequently.  A last resort is to
	# sleep for some time.
	start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
	[ "$?" = 2 ] && return 2
	# Many daemons dont delete their pidfiles when they exit.
	rm -f $PIDFILE
	return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
	#
	# If the daemon can reload its configuration without
	# restarting (for example, when it is sent a SIGHUP),
	# then implement that here.
	#
	start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
	return 0
}

case "$1" in
  start)
	[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  stop)
	[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  #reload|force-reload)
	#
	# If do_reload() is not implemented then leave this commented out
	# and leave force-reload as an alias for restart.
	#
	#log_daemon_msg "Reloading $DESC" "$NAME"
	#do_reload
	#log_end_msg $?
	#;;
  restart|force-reload)
	#
	# If the "reload" option is implemented then remove the
	# "force-reload" alias
	#
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case "$?" in
	  0|1)
		do_start
		case "$?" in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
		;;
	  *)
	  	# Failed to stop
		log_end_msg 1
		;;
	esac
	;;
  *)
	#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
	echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
	exit 3
	;;
esac

:

Hi!
Works fine for me.

Is it possible, that platform hooks won't be executed, if elastic beanstalk scales?
If I run 'eb scale 5', 4 additional ec2 instances are running but I can see from qmonitor, that only one ec2 instance runs the qcluster.

If I scale back to one ec2 instance, the instance that is actually running may get shut down.

Someone knows any solution to that?

Hi @EdmundoDelGusto, that's a good question. I didn't encounter it but have a few suggestions.
Firstly, there's a way to ensure command execution restricted to "leader_only" instance of the autoscaling group.

For .sh files

if [[ "$EB_IS_COMMAND_LEADER" == "true" ]]; then
  COMMANDS
fi

Or as an alternative commands which are triggered from .ebextensions folder within somename.config file:

container_commands:
  name of container_command:
    command: "command to run"
    leader_only: true

Secondly, there's also a shortcut to skip all supervisor configuration and create a Procfile in the app root. I'm using it now on Heroku for running django-q worker process. I have no idea, how it will behave in Beanstalk, but it definitely worth a try.

web: gunicorn <coreapp>.wsgi
worker: python manage.py qcluster --settings=<coreapp>.settings

, where <coreapp> - is your django app name with settings module.

Configuring the WSGI server with a Procfile
Extending Elastic Beanstalk Linux platforms

I don't have a sandbox beanstalk to test autoscaling, so I'd appreciate if you share your results.

Thank you for all the help.

I think I messed up when I additionally added sudo yum install supervisor to the container_commands. I am not sure. But it looked like this was causing the other instances to fail. (I'm fairly inexperienced with elastic beanstalk). But now, if I scale up, django q runs on all instances.

I got some additional problems that someone may also run into:

1. When creating the scripts with Windows, you need to remove the \r symbols Windows adds.

I've done this in my .ebextensions/django.config

  009_remove_windows_symbols:
    command: "sed -i 's/\r$//' .platform/hooks/postdeploy/01_set_env.sh"
  010_remove_windows_symbols:
    command: "sed -i 's/\r$//' .platform/hooks/postdeploy/02_django_q.sh"
  011_remove_windows_symbols:
    command: "sed -i 's/\r$//' .platform/hooks/postdeploy/03_supervisor_init.sh"
  012_remove_windows_symbols:
    command: "sed -i 's/\r$//' supervisord.sample"

(This probably could be done more efficiently)

2. I needed to make all scripts executable in github.

git add --chmod=+x -- .platform/*/*/*.sh 
git add --chmod=+x -- *.sample 

(I am not sure if all operations are necessary)

Open Problem

One open problem is, that I think the last script (04) does not do anything for me.
If I log into the instance with eb ssh and run sudo service supervisord stop nothing happens. Same for all other commands. No output whatsoever and django_q keeps running.

Beanstalk supports processes configuration through Procfile file in your project root.

I strongly recommend to replace this gist configuration with Procfile worker description.
Sample with gunicorn server:

web: gunicorn --bind :8000 --workers 3 --threads 2 yourapp.wsgi:application
worker: python manage.py qcluster --settings=yourapp.settings