Celery hang problem

problem.md

Celery 4.1.0, Redis client 2.10.6, Redis server 4.0

I have a problem that seems to be related to synchronous subtasks, except that none of the workarounds (or indeed the errors/warnings that I should expect from the docs) are having any effect.

I have 3 worker servers, A B and C

• A and B run the same codebase.
• A calls apply_async of a Task1. Get/Forget not called (it doesn't care about the result itself, but we want the result stored in the backend)
• Task1 gets run by server B.
• Within Task1 (now executed on server B) - Task2 from another codebase is called - using send_task (since the codebase is not shared here).
• .get() is called on the AsyncResult of Task2
• Server C picks up Task2 and runs it.
• Task2 finishes.
• Task1 hangs waiting for the .get() of Task2

The problem is that inside Task1 I'm calling .get() on the AsyncResult given by calling send_task on Task2 - this seems to hang forever even though Task2 completes (fails, in my test case, but still, the result is there in the backend [redis], and I can see it in Flower) -- Task1 never completes as it is blocked waiting on the get() of Task2. I have tried disable_sync_subtasks=False, and with allow_join_result() -- though I have never seen the exception/warning mentioned and indeed the subtask gets sent and run anyway.

tasks.py

# Process on Server A, running hourly
def do_a_thing():
    """Runs hourly and sends this task."""
    task_1.apply_async(
        kwargs={'kwarg1': 123},
        queue='somequeue',
        routing_key='somequeue'
        eta=eta
    )

# Task1 Code, runs on Server B:
def task_1(kwarg1=None):
    """Run some other task handled by another server/codebase entirely."""
    task2 = celery_app.send_task(
        'other.codebase.messaging.task_2',
        kwargs={'kwarg2': kwarg1},
    )
    
    # Code hangs here, despite task2 completing on server C:
    # with allow_join_result():  # uncommenting this has no effect
    result = task2.get(
        # disable_sync_subtasks=False,  # uncommenting this has no effect
    )
    
    # Code below never executed
    if result is None:
        raise ValueError()