Last active
August 5, 2023 21:36
-
-
Save yadav-vikas/cd79eb8c59f9030fdafc202783a08296 to your computer and use it in GitHub Desktop.
How to integrate Celery and RabbitMQ as systemd service in Linux
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Name of nodes to start | |
# here we have a single node | |
CELERYD_NODES="w1 w2" | |
# or we could have three nodes: | |
# CELERYD_NODES="w1 w2 w3" | |
# Absolute or relative path to the 'celery' command: | |
#CELERY_BIN="/usr/local/bin/celery" | |
CELERY_BIN="/home/$USER/project_stack/project_name/venv/bin/celery" | |
# App instance to use | |
# comment out this line if you don't use an app | |
CELERY_APP="project_name" | |
# or fully qualified: | |
#CELERY_APP="proj.tasks:app" | |
# How to call manage.py | |
CELERYD_MULTI="multi" | |
# Extra command-line arguments to the worker | |
CELERYD_OPTS="--time-limit=300 --concurrency=2" | |
# - %n will be replaced with the first part of the nodename. | |
# - %I will be replaced with the current child process index | |
# and is important when using the prefork pool to avoid race conditions. | |
# CELERYD_PID_FILE="/var/run/celery/%n.pid" | |
# CELERYD_LOG_FILE="/var/log/celery/%n%I.log" | |
CELERYD_PID_FILE="/var/run/celery/celery.pid" | |
CELERYD_LOG_FILE="/var/log/celery/celery.log" | |
CELERYD_LOG_LEVEL="INFO" | |
# you may wish to add these options for Celery Beat | |
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid" | |
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Unit] | |
Description=Celery Service | |
After=network.target | |
[Service] | |
Type=forking | |
User=$USER | |
Group=$USER | |
EnvironmentFile=/etc/opt/celery.conf | |
WorkingDirectory=/home/$USER/project_stack/project_name | |
ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \ | |
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ | |
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' | |
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \ | |
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ | |
--loglevel="${CELERYD_LOG_LEVEL}"' | |
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \ | |
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ | |
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' | |
Restart=always | |
[Install] | |
WantedBy=multi-user.target |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Unit] | |
Description=Celery Beat Service | |
After=network.target | |
[Service] | |
Type=simple | |
User=$USER | |
Group=$USER | |
EnvironmentFile=/etc/opt/celery.conf | |
WorkingDirectory=/home/$USER/project_stack/project_name | |
ExecStart=/bin/sh -c '${CELERY_BIN} -A ${CELERY_APP} beat \ | |
--pidfile=${CELERYBEAT_PID_FILE} \ | |
--logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}' | |
Restart=always | |
[Install] | |
WantedBy=multi-user.target |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
###################### please follow these steps to configure Celery & RabbitMQ ###################### | |
#1. copy `install-rabbitmq.sh` on your system then change permissions of the file using `sudo chmod +x install-rabbitmq.sh` | |
and finally install RabbitMQ on your system by running `./install-rabbitmq.sh` . | |
#2. install `Celery` on the your python application and copy the venv/bin path location | |
#3. paste `celery.service` at /etc/systemd/system/ and update the file | |
#4. paste `celerybeat.service` at /etc/systemd/system/ and update the file | |
>>> Note: Please change user, group, WorkingDirectory and CELERY_BIN in celery.service, celerybeat.service and celery.conf files respectively based on your project folder directory location | |
#5. paste `celery.conf` at /etc/opt/ | |
#6. once done, run below commands. | |
sudo mkdir -pv /var/{log,run}/celery/ | |
sudo chown -cR $USER:$USER /var/{log,run}/celery/ | |
sudo systemctl enable celery.service | |
sudo systemctl enable celerybeat.service | |
sudo systemctl start celery.service | |
sudo systemctl start celerybeat.service | |
sudo systemctl daemon-reload | |
sudo systemctl status celery.service |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
##### please make this file executible to run the following configuraions ###### | |
# you can run the following... | |
# sudo chmod +x install-rabbitmq.sh | |
############################################################################### | |
sudo apt update && sudo apt upgrade | |
sudo apt install curl wget gnupg apt-transport-https -y | |
curl -fsSL https://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc | sudo gpg --dearmor -o /usr/share/keyrings/erlang.gpg | |
echo "deb [signed-by=/usr/share/keyrings/erlang.gpg] https://packages.erlang-solutions.com/ubuntu $(lsb_release -cs) contrib" | sudo tee /etc/apt/sources.list.d/erlang.list | |
sudo apt update | |
sudo apt install erlang -y | |
curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.deb.sh | sudo bash | |
sudo apt install rabbitmq-server -y | |
sudo systemctl enable rabbitmq-server.service | |
sudo systemctl restart rabbitmq-server.service |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment