celery.md

This post is targetted towards Celery beginners. It discusses different ways to run Celery.

• Using celery with a single module
• Using celery with multiple modules.
• Using celery with multiple modules split across packages.

Prerequisites

You should have Celery and Redis installed. We will use Redis as our broker.

Using celery with single module.

Let's create a module called tasks.py with following content:

from celery import Celery

app = Celery('tasks', broker='redis://localhost')

@app.task
def hello():
    return 'hello'

Let's start the Celery worker

celery -A tasks worker -l INFO

Let's start a shell and queue a task.

In [1]: from tasks import hello

In [2]: hello.apply_async()
Out[2]: <AsyncResult: 079bd2b9-05da-4660-9064-b369b3863cb0>

You should see a message on the worker's console. This verifies that Celery worker processed the task.

Using celery with multiple modules.

Let's create a module called mathematics.py with following content.

from tasks import app

@app.task
def sum(a, b):
    return a + b

We need to ask the Celery app instance to include this module so that when worker is started it is aware about this module's tasks.

So, app instantiation in tasks.py need to be modified to look like the following:

app = Celery('tasks', broker='redis://localhost', include=['mathematics'])

Restart the celery worker.

Restart python shell and queue the task sum.

In [2]: from mathematics import sum

In [4]: sum.apply_async(args=(3, 4))
Out[4]: <AsyncResult: 1ba8aea0-a604-49d6-aa45-a0315d2b58c6>

You should see the computed sum on the worker's console. This verifies that Celery worker processed the task.

Using celery in a multi package setup

Let's create a package called internationalization.

╰─$ mkdir internationalization

╰─$ touch internationalization/__init__.py

Let's create modules german.py and french.py in it.

internationalization/german.py

# internationalization/german.py
from tasks import app

@app.task
def hello():
    return "Hallo"

internationalization/french.py

# internationalization/french.py

from tasks import app

@app.task
def hello():
    return 'Bonjour'

We need to include these task modules in our Celery app. Modify app in tasks.py so it looks like:

app = Celery('tasks', broker='redis://localhost', include=['mathematics', 'internationalization.french', 'internationalization.german'])

Restart the celery worker.

Restart python shell and queue the tasks hello from german.py and french.py

In [1]: from internationalization.french import hello

In [2]: hello.apply_async()
Out[2]: <AsyncResult: 09297a37-2733-4820-8f37-cd6eeeb5ce5b>

You should see the message on the worker's console. This verifies that Celery worker processed the task.