Skip to content

Canvas with group-task-task-group does not work #5354

Description

@AbdealiLoKo

Checklist

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have included all related issues and possible duplicate issues in this issue.
  • I have included the contents of pip freeze in the issue.
  • I have verified that the issue exists against the master branch of Celery.
  • I have tried reproducing the issue on more than one message broker and/or result backend.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with retries, ETA/Countdown & rate limits disabled.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version: 4.2.1 (windowlicker)

celery report Output:
software -> celery:4.2.1 (windowlicker) kombu:4.2.1 py:3.6.4
            billiard:3.5.0.4 redis:2.10.6
platform -> system:Darwin arch:64bit imp:CPython
settings -> transport:redis results:redis://localhost/

BROKER_URL: 'redis://localhost:6379//'
CELERY_RESULT_BACKEND: 'redis://localhost/'
BROKER_TRANSPORT_OPTIONS: {
 'visibility_timeout': 11700}

Tried this with RabitMQ also. Same issue exists.

Steps to Reproduce

If I create a celery canvas with a task - grouptask - task - task - grouptask - task, it fails.
If I change it to task - grouptask - task - grouptask - task, it works

Minimally Reproducible Test Case

    @celery_app.task
    def print_task(val):
        print("Running print_task. val is: {}".format(val))

    final_task = celery.chain(
        print_task.si(val="Starter task"),
        celery.group([
            print_task.si(val='group1, task1'),
            print_task.si(val='group1, task2'),
        ]),
        print_task.si(val="group1, after-task"),
        print_task.si(val="group1, after-task2"),  # <---- commenting this makes it work
        celery.group([
            print_task.si(val='group2, task1'),
            print_task.si(val='group2, task2'),
        ]),
        print_task.si(val="FINAL task"),
    )
    r = final_task()
    print(r.get())

Also, when we do a print(final_task), it shows:

print_task(val='Starter task')
| %(print_task([
		print_task(val='group1, task1'),
		print_task(val='group1, task2')
	], val='group1, after-task')
	| print_task(val='group1, after-task2')
) | %print_task([
		print_task(val='group2, task1'),
		print_task(val='group2, task2')
	], val='FINAL task')

Required Dependencies

  • Minimal Python Version: Python 3.6.4 :: Anaconda, Inc.
  • Minimal Broker Version: Redis 4.0.9
  • Minimal Result Backend Version: Redis 4.0.9
  • Minimal OS and/or Kernel Version: Mac OSX 10.13, Linux CentOS7
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown

I've also tried with latest git master versions of celery, kombu, amqp, redis-py, vine.

Expected Behavior

The output should have executed the entire canvas:

Running print_task. val is: Starter task
Running print_task. val is: group1, task1
Running print_task. val is: group1, task2
Running print_task. val is: group1, after-task
Running print_task. val is: group1, after-task2
Running print_task. val is: group2, task1
Running print_task. val is: group2, task2
Running print_task. val is: FINAL task

Actual Behavior

The tasks until group1, after-task2 work fine. The group1, after-task2 also completes execution according to celery logs.

Running print_task. val is: Starter task
Running print_task. val is: group1, task1
Running print_task. val is: group1, task2
Running print_task. val is: group1, after-task
Running print_task. val is: group1, after-task2
[2019-02-22 16:30:18,041: INFO/MainProcess] Task print_task[beaea403-5f2e-4b84-8f64-bcada7c07000] succeeded in 0.004693654947914183s: None

The redis-cli shows that is completed too:

127.0.0.1:6379> get celery-task-meta-beaea403-5f2e-4b84-8f64-bcada7c07000
"{\"status\": \"SUCCESS\", \"result\": null, \"traceback\": null, \"children\": [], \"task_id\": \"beaea403-5f2e-4b84-8f64-bcada7c07000\"}"

But for some reason, the children is empty - even though there are children.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions