Skip to content

Instantly share code, notes, and snippets.

@victorcarrico
Created December 4, 2018 18:58
Show Gist options
  • Save victorcarrico/78aa25c84de59cb6ac838ed08b659eb1 to your computer and use it in GitHub Desktop.
Save victorcarrico/78aa25c84de59cb6ac838ed08b659eb1 to your computer and use it in GitHub Desktop.
Celery Canvas: It's basically the inverse chord which is like a map_reduce. This run a task and with the result run a celery group with the results.
from celery import task, subtask, group
@task
def get_list(amount):
return [i for i in range(amount)]
@task
def process_item(item):
# do stuff
pass
@task
def dmap(it, callback):
# Map a callback over an iterator and return as a group
callback = subtask(callback)
return group(callback.clone([arg,]) for arg in it)()
# runs process_item for each item in the return of get_list
process_list = (get_list.s(10) | dmap.s(process_item.s()))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment