Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Django, Postgres 9.6 and Celery on Elastic Beanstalk - ebextensions
packages:
yum:
libjpeg-turbo-devel: []
libpng-devel: []
libcurl-devel: []
commands:
01_install_rhel_pg:
command: "(yum repolist |grep -q pgdg96) || sudo yum install https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-6-x86_64/pgdg-ami201503-96-9.6-2.noarch.rpm -y"
02_install_pg_devel:
command: "sudo yum install postgresql96-devel -y"
03_link_pg_config:
command: "sudo ln -sf /usr/pgsql-9.6/bin/pg_config /usr/bin/"
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh" :
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}
# Create celery configuration script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A assembly --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A assembly --loglevel=INFO --workdir=/tmp -S django
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
commands:
01_upgrade_pip_global:
command: "if test -e /usr/bin/pip; then sudo /usr/bin/pip install --upgrade pip; fi"
02_upgrade_pip_global:
command: "if test -e /usr/local/bin/pip; then sudo /usr/local/bin/pip install --upgrade pip; fi"
03_upgrade_pip_for_venv:
command: "if test -e /opt/python/run/venv/bin/pip; then sudo /opt/python/run/venv/bin/pip install --upgrade pip; fi"
container_commands:
02_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
03_migrate:
command: "./manage.py migrate"
leader_only: true
option_settings:
aws:elasticbeanstalk:container:python:
WSGIPath: assembly/wsgi.py
aws:elasticbeanstalk:application:environment:
DJANGO_SETTINGS_MODULE: django_app.settings

This configuration will run celery-worker and celery-beat on a single (leader) node, using supervisor.

Also it'll run './manage.py migrate' on a single node after each deployment

Ensure that requirements.txt includes:

pycurl==7.43.0 --global-option="--with-nss"

You may also need to do:

eb setenv PYCURL_SSL_LIBRARY=nss

Note that you should not include AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your settings. Boto will pick them up from the instance profile automatically.

Ensure your instance profile (aws-elasticbeanstalk-ec2-role) has permission to talk to SQS, and possibly SES if you use django-ses.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:*"
            ],
            "Resource": "arn:aws:sqs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ses:SendRawEmail",
                "ses:SendEmail",
                "ses:GetSendQuota"
            ],
            "Resource": "*"
        }
    ]
}

Much of this was sourced from:

https://stackoverflow.com/questions/41161691/how-to-run-a-celery-worker-with-django-app-scalable-by-aws-elastic-beanstalk

SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY')
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
CELERY_TASK_SERIALIZER='json'
CELERY_RESULT_BACKEND = 'django-db'
CELERY_BROKER_URL = 'sqs://'
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "us-west-2",
'queue_name_prefix': 'myapp-something-',
'visibility_timeout': 360,
'polling_interval': 1
}
# Use django-ses for email
EMAIL_BACKEND = 'django_ses.SESBackend'
AWS_SES_REGION_ENDPOINT = 'email.us-west-2.amazonaws.com'
@simple-scout

This comment has been minimized.

Copy link

@simple-scout simple-scout commented May 15, 2019

Thanks for sharing this! I'm having a heck of a time getting celery to work correctly on elastic beanstalk.

I'm currently having trouble getting pycurl installed correctly. My worker log shows that the compile time and runtime PYCURL_SSL_LIBRARY are different. In your notes, you state that you may need to run eb setenv PYCURL_SSL_LIBRARY=nss. Is this done before eb deploy?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment