This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import logging | |
| from celery.app.defaults import DEFAULT_TASK_LOG_FMT, DEFAULT_PROCESS_LOG_FMT | |
| class CeleryTaskFilter(logging.Filter): | |
| def filter(self, record): | |
| return record.processName.find('Worker') != -1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Celery base task aimed at longish-running jobs that return a result. | |
| ``AwesomeResultTask`` adds thundering herd avoidance, result caching, progress | |
| reporting, error fallback and JSON encoding of results. | |
| """ | |
| from __future__ import division | |
| import logging | |
| import simplejson |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| '''Zookeeper-based Scheduler''' | |
| ## Standard Library | |
| import cPickle # Store dictionary in ZooKeeper | |
| import datetime # Time delta | |
| import socket # Hostname | |
| ## Third Party | |
| import celery # Current app | |
| import celery.beat # Scheduler | |
| import celery.utils.log # Get logger | |
| import kazoo.client # ZooKeeper Client Library |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from celery import Task | |
| from celery.task import task | |
| from my_app.models import FailedTask | |
| from django.db import models | |
| @task(base=LogErrorsTask) | |
| def some task(): | |
| return result | |
| class LogErrorsTask(Task): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pickle | |
| import threading | |
| from Queue import Queue | |
| import time | |
| from bson import InvalidDocument | |
| from celery.utils.log import get_task_logger | |
| logger = get_task_logger(__name__) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from __future__ import absolute_import, division, print_function | |
| import multiprocessing | |
| import pickle | |
| from multiprocessing.pool import ThreadPool | |
| from celery import shared_task | |
| from dask.local import get_async # TODO: get better get | |
| from dask.context import _globals | |
| from dask.optimize import fuse, cull |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import time | |
| from memory_profiler import memory_usage | |
| import logging | |
| celery_logger = logging.getLogger('celery') | |
| def track_celery(method): | |
| """ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import os | |
| import sys | |
| from kombu import Exchange, Queue | |
| from datetime import timedelta | |
| from config import cfg | |
| import log |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| https://docs.celeryproject.org/en/stable/tutorials/task-cookbook.html | |
| https://github.com/channeng/celery-scheduler | |
| https://medium.com/@yedjoe/celery-4-periodic-task-in-django-9f6b5a8c21c7 | |
| https://medium.com/@contrapasso/how-i-got-faang-offers-without-grinding-leetcode-7e556243e9ce | |
| https://appliku.com/post/django-celery-tutorial-to-background-tasks | |
| https://docs.celeryproject.org/en/stable/django/first-steps-with-django.html | |
| https://idego-group.com/grab-your-free-book-with-django-and-celery/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # This work is licensed under the GNU Public License (GPL). | |
| # To view a copy of this license, visit http://www.gnu.org/copyleft/gpl.html | |
| # For more information visit this blog post http://mpcabd.igeex.biz/python-celery-asynchronous-task-decorator/ | |
| # Written by Abd Allah Diab (mpcabd) | |
| # Email: mpcabd ^at^ gmail ^dot^ com | |
| # Website: http://mpcabd.igeex.biz | |
| from django.utils.decorators import available_attrs |