Skip to content

group([chain(group(task(), task()), group(...)), chain(group(...), group(...))]) construct never finishes #2354

Description

@mittagessen

I've got several chains containing groups which should run in parallel. As far as I can tell these chains run perfectly well (celery -l DEBUG and generated output on the file system) but the ready() function of the GroupResult and AsyncResults returns always false. Unfortunately, the chains' children fields are empty making debugging what is happening inside the inner groups quite hard.

A very simple test case displaying this behavior:

from celery import Celery

app = Celery(main='tasks', broker='redis://')
app.config_from_object(celeryconfig)

@app.task(name=u'A')
def A():
    print('A')

A test case that fails:

>>> from celery import group, chain
>>> chain1 = chain(group([tasks.A.si(), tasks.A.si()]), group([tasks.A.si(), tasks.A.si()]))
>>> r = group([chain1, chain1])()
>>> r.save()
...
>>> r.ready()
False
>>> r.results[0].ready()
False

On the other hand if the interior group contains just a single task the return values are as expected:

>>> chain1 = chain(group([tasks.A.si()]), group([tasks.A.si()]))
>>> r = group([chain1, chain1])()
>>> r.save()
>>> r.ready()
True
>>> r.results[0].ready()
True

I've tried running this example against 3.1.16 an the current git master branch using the redis message broker:

software -> celery:3.1.16 (Cipater) kombu:3.0.23 py:2.7.8
            billiard:3.3.0.18 redis:2.10.3
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://127.0.0.1:6379

BROKER_URL: 'redis://127.0.0.1:6379//'
CELERY_RESULT_SERIALIZER: 'json'
CELERY_ENABLE_UTC: True
CELERY_TIMEZONE: 'Europe/Berlin'
CELERY_TASK_SERIALIZER: 'json'
CELERY_RESULT_BACKEND: 'redis://127.0.0.1:6379'

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions