Bash script to install Django Q with Supervisor on Amazon Linux 2 (AL2 / Elastic Beanstalk) - UPD with Procfile in the comments

01_set_env.sh

#!/bin/bash

#Create a copy of the environment variable file.
sudo cp /opt/elasticbeanstalk/deployment/env /opt/elasticbeanstalk/deployment/custom_env_var

#Set permissions to the custom_env_var file so this file can be accessed by any user on the instance. You can restrict permissions as per your requirements.
sudo chmod 644 /opt/elasticbeanstalk/deployment/custom_env_var

#Remove duplicate files upon deployment.
sudo rm -f /opt/elasticbeanstalk/deployment/*.bak

#Install/enable extra packages
sudo amazon-linux-extras install epel -y

#Install supervisor
sudo yum install supervisor -y >/dev/null 2>&1
if [ $? -ne 1 ]; then   # Exit on any any error except 'nothing to do' 
    exit 0
fi

02_django_migrate.sh

#!/bin/bash

source /var/app/venv/*/bin/activate && {

# collect static
python manage.py collectstatic --noinput;
# log which migrations have already been applied
python manage.py showmigrations;
# migrate user model prior to other models if you have Custom User Model < NB
#python manage.py migrate users --noinput;
# migrate the rest
python manage.py migrate --noinput;
# create superuser
python manage.py createsu;

}

03_django_q.sh

#!/usr/bin/env bash

# Author: Valentine Solonechnyi <valentinesolo@gmail.com>
# Based on instructions by Rémi Héneault 
# https://gist.github.com/codeSamuraii/0e11ce6d585b3290b15a9ad163b9aa06
# Django Q supervisor configuration and setup for Amazon Linux 2

mkdir -p /var/log/djangoq/ /var/run/djangoq/

# Get django environment variables
# grep '^PYTHONPATH\|^PATH' no filtering of env variables
# djangoqenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '\n' ',' | sed 's/=/="/'g | sed 's/,/",/g'`
# fix from @matiszz
djangoqenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '\n' ',' | sed 's/=/="/'g | sed 's/,/",/g' | sed 's/="="/=="/'g | sed 's/""/"/'g`
djangoqenv=${djangoqenv%?}

# Create djangoq configuraiton script
djangoqconf="[program:django-q]
command=bash -c 'source /var/app/venv/*/bin/activate && python manage.py qcluster'
directory=/var/app/current
user=nobody
numprocs=1
stdout_logfile=/var/log/djangoq/worker.log
stderr_logfile=/var/log/djangoq/worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
stopasgroup=true
environment=$djangoqenv
"

# Create the djangoq supervisord conf script
echo "$djangoqconf" | sudo tee /etc/supervisord.d/djangoq.conf

# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "files = supervisord.d/*.conf" /etc/supervisord.conf
  then
  sed -i "s/*.ini/*.conf/g" /etc/supervisord.conf
fi

#Launch supervisord process
sudo supervisord -c /etc/supervisord.conf

# Reread the supervisord config
sudo supervisorctl reread

# Update supervisord in cache without restarting all services
sudo supervisorctl update

# Start/Restart djangoqd through supervisord.
sudo supervisorctl -c /etc/supervisord.conf restart django-q

04_supervisor_init.sh

#!/usr/bin/env bash

#Supervisor init config

# Add a new configuration of restart Supervisor
sudo cp /var/app/current/supervisord.sample /etc/init.d/supervisord

#Add execute authority
sudo chmod +x /etc/init.d/supervisord

#Add the configuration into system
sudo chkconfig --add supervisord

#Switch on the configuration and start
sudo chkconfig supervisord on
sudo service supervisord start

supervisord.sample

#! /bin/sh
### BEGIN INIT INFO
# Provides:          supervisord
# Required-Start:    $remote_fs
# Required-Stop:     $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Example initscript
# Description:       This file should be used to construct scripts to be
#                    placed in /etc/init.d.
### END INIT INFO

# Author: Dan MacKinlay <danielm@phm.gov.au>
# Based on instructions by Bertrand Mathieu
# http://zebert.blogspot.com/2009/05/installing-django-solr-varnish-and.html

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Description of the service"
NAME=supervisord
DAEMON=/usr/local/bin/supervisord
DAEMON_ARGS=""
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
	# Return
	#   0 if daemon has been started
	#   1 if daemon was already running
	#   2 if daemon could not be started
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
		|| return 1
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
		$DAEMON_ARGS \
		|| return 2
	# Add code here, if necessary, that waits for the process to be ready
	# to handle requests from services started subsequently which depend
	# on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
    # Return
	#   0 if daemon has been stopped
	#   1 if daemon was already stopped
	#   2 if daemon could not be stopped
	#   other if a failure occurred
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
	RETVAL="$?"
	[ "$RETVAL" = 2 ] && return 2
	# Wait for children to finish too if this is a daemon that forks
	# and if the daemon is only ever run from this initscript.
	# If the above conditions are not satisfied then add some other code
	# that waits for the process to drop all resources that could be
	# needed by services started subsequently.  A last resort is to
	# sleep for some time.
	start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
	[ "$?" = 2 ] && return 2
	# Many daemons dont delete their pidfiles when they exit.
	rm -f $PIDFILE
	return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
	#
	# If the daemon can reload its configuration without
	# restarting (for example, when it is sent a SIGHUP),
	# then implement that here.
	#
	start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
	return 0
}

case "$1" in
  start)
	[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  stop)
	[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  #reload|force-reload)
	#
	# If do_reload() is not implemented then leave this commented out
	# and leave force-reload as an alias for restart.
	#
	#log_daemon_msg "Reloading $DESC" "$NAME"
	#do_reload
	#log_end_msg $?
	#;;
  restart|force-reload)
	#
	# If the "reload" option is implemented then remove the
	# "force-reload" alias
	#
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case "$?" in
	  0|1)
		do_start
		case "$?" in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
		;;
	  *)
	  	# Failed to stop
		log_end_msg 1
		;;
	esac
	;;
  *)
	#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
	echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
	exit 3
	;;
esac

:

This is great, thanks for sharing! I was wondering when do you run these/what is your file structure like? Are they done in .platform/hooks/postdeploy?

You're right, sh files 01-04 are placed in .platform/hooks/postdeploy and
supervisord.sample I've put into the app root folder for convinience. It might be placed anywhere as long as this path is correct
(sudo cp /var/app/current/supervisord.sample /etc/init.d/supervisord)

https://valentine.click/blog/django-q-and-beanstalk - I've added some comments in this blog post.

Thanks a lot, this was very useful! A couple of problems I faced that hopefully can be useful for someone else.

• First of all, remember to add +x (execution) permission to these files by adding this command to a file your_project/.ebextensions/01_permissions.config:

container_commands:
    01_chmod1:
        command: "chmod +x .platform/hooks/postdeploy/*"

• Then I had some problems as my env variables were containing some strings with two equal signs (==). It took me a while to understand that that was causing the problem. I had to modify 03_django_q.sh - Line 12 for this:

djangoqenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '\n' ',' | sed 's/=/="/'g | sed 's/,/",/g' | sed 's/="="/=="/'g | sed 's/""/"/'g`

The old script was changing all equal signs (=) by =", so since my env variable has some keys that contain double equal, it was messing everything. Please note that this is my first time using sed so it can surely be done better. But it works!

Hi!
Works fine for me.

Is it possible, that platform hooks won't be executed, if elastic beanstalk scales?
If I run 'eb scale 5', 4 additional ec2 instances are running but I can see from qmonitor, that only one ec2 instance runs the qcluster.

If I scale back to one ec2 instance, the instance that is actually running may get shut down.

Someone knows any solution to that?

Hi @EdmundoDelGusto, that's a good question. I didn't encounter it but have a few suggestions.
Firstly, there's a way to ensure command execution restricted to "leader_only" instance of the autoscaling group.

For .sh files

if [[ "$EB_IS_COMMAND_LEADER" == "true" ]]; then
  COMMANDS
fi

Or as an alternative commands which are triggered from .ebextensions folder within somename.config file:

container_commands:
  name of container_command:
    command: "command to run"
    leader_only: true

Secondly, there's also a shortcut to skip all supervisor configuration and create a Procfile in the app root. I'm using it now on Heroku for running django-q worker process. I have no idea, how it will behave in Beanstalk, but it definitely worth a try.

web: gunicorn <coreapp>.wsgi
worker: python manage.py qcluster --settings=<coreapp>.settings

, where <coreapp> - is your django app name with settings module.

Configuring the WSGI server with a Procfile
Extending Elastic Beanstalk Linux platforms

I don't have a sandbox beanstalk to test autoscaling, so I'd appreciate if you share your results.

Thank you for all the help.

I think I messed up when I additionally added sudo yum install supervisor to the container_commands. I am not sure. But it looked like this was causing the other instances to fail. (I'm fairly inexperienced with elastic beanstalk). But now, if I scale up, django q runs on all instances.

I got some additional problems that someone may also run into:

1. When creating the scripts with Windows, you need to remove the \r symbols Windows adds.

I've done this in my .ebextensions/django.config

  009_remove_windows_symbols:
    command: "sed -i 's/\r$//' .platform/hooks/postdeploy/01_set_env.sh"
  010_remove_windows_symbols:
    command: "sed -i 's/\r$//' .platform/hooks/postdeploy/02_django_q.sh"
  011_remove_windows_symbols:
    command: "sed -i 's/\r$//' .platform/hooks/postdeploy/03_supervisor_init.sh"
  012_remove_windows_symbols:
    command: "sed -i 's/\r$//' supervisord.sample"

(This probably could be done more efficiently)

2. I needed to make all scripts executable in github.

git add --chmod=+x -- .platform/*/*/*.sh 
git add --chmod=+x -- *.sample 

(I am not sure if all operations are necessary)

Open Problem

One open problem is, that I think the last script (04) does not do anything for me.
If I log into the instance with eb ssh and run sudo service supervisord stop nothing happens. Same for all other commands. No output whatsoever and django_q keeps running.

Beanstalk supports processes configuration through Procfile file in your project root.

I strongly recommend to replace this gist configuration with Procfile worker description.
Sample with gunicorn server:

web: gunicorn --bind :8000 --workers 3 --threads 2 yourapp.wsgi:application
worker: python manage.py qcluster --settings=yourapp.settings