My first shot at fixing this was in tasks.py file below.
But then that just gave a "unhandled error in deferred", so I went on to use CrawlRunner.
That showed no output at all anymore, and didn't run as expected.
Eventually, I just settled on CELERY_WORKER_MAX_TASKS_PER_CHILD=1=1 in settings.py
Note: CELERY_WORKER_MAX_TASKS_PER_CHILD=1 is for django. Celery without django probably drops the CELERY_ prefix