Skip to content

RabbitMQ detects cycle and drops message when using quorum queues with delayed tasks + retry #9867

Description

@daniellang-dt

Checklist

  • I have verified that the issue exists against the main branch of Celery.
  • This has already been asked to the discussions forum first.
  • I have read the relevant section in the
    contribution guide
    on reporting bugs.
  • I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the main branch.
  • I have included all related issues and possible duplicate issues
    in this issue (If there are none, check this box anyway).
  • I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the main branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version: 5.5.3

celery report Output:

software -> celery:5.5.3 (immunity) kombu:5.6.0b2 py:3.12.8
            billiard:4.2.1 py-amqp:5.3.1
platform -> system:Linux arch:64bit, ELF
            kernel version:6.8.0-51-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:disabled

deprecated_settings: None
broker_url: 'amqp://user:********@localhost:5672/my_host'
task_default_exchange_type: 'topic'
task_queues: [<unbound Queue celery-quorum -> <unbound Exchange celery-quorum(topic)> -> celery-quorum>]
task_default_queue: 'celery-quorum'
broker_transport_options: 
 'confirm_publish': True}

Steps to Reproduce

Required Dependencies

  • Minimal Python Version: N/A or Unknown
  • Minimal Celery Version: N/A or Unknown
  • Minimal Kombu Version: N/A or Unknown
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: RabbitMQ 3.13.7 and 4.1.2
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.3.1  
billiard==4.2.1  
boto3==1.40.13  
botocore==1.40.13  
celery @ git+https://github.com/celery/celery@33eb14852310996b1909c8388cd319809d6c8626  
certifi==2025.8.3  
charset-normalizer==3.4.3  
click==8.2.1  
click-didyoumean==0.3.1  
click-plugins==1.1.1.2  
click-repl==0.3.0  
debugpy==1.8.16  
docker==7.1.0  
idna==3.10  
iniconfig==2.1.0  
jmespath==1.0.1  
kombu==5.6.0b2  
packaging==25.0  
pluggy==1.6.0  
prompt-toolkit==3.0.51  
psutil==7.0.0  
pygments==2.19.2  
pytest==8.4.1  
pytest-celery==1.2.1  
pytest-docker-tools==3.1.9  
python-dateutil==2.9.0.post0  
python-dotenv==1.1.1  
python-memcached==1.62  
redis==6.4.0  
requests==2.32.5  
s3transfer==0.13.1  
setuptools==80.9.0  
six==1.17.0  
tenacity==9.1.2  
tzdata==2025.2  
urllib3==1.26.20  
vine==5.1.0  
wcwidth==0.2.13

Other Dependencies

Details

N/A

Minimally Reproducible Test Case

Details

# celeryconfig.py
RABBITMQ_USERNAME = os.getenv('RABBITMQ_USERENAME', 'guest')
RABBITMQ_PASSWORD = os.getenv('RABBITMQ_PASSWORD', 'guest')

app = Celery('celery_test')

app.conf.update(
    broker_url=f'pyamqp://{RABBITMQ_USERNAME}:{RABBITMQ_PASSWORD}@localhost:5672/my_vhost',
    task_default_exchange_type='topic',
    task_queues=[
        Queue(
            'celery-quorum', 
            exchange=Exchange('celery-quorum', type='topic'),
            routing_key='celery-quorum',
            queue_arguments={'x-queue-type': 'quorum'}
        ),
    ],
    task_default_queue='celery-quorum',
    broker_transport_options={"confirm_publish": True},
)

app.autodiscover_tasks(['tasks'])


# tasks.py
@app.task(autoretry_for=(ValueError,), retry_kwargs={"max_retries": 2},default_retry_delay=15)
def dummy_task():
    print("Dummy task is running")
    raise ValueError("Dummy task is broken")

# client.py
from tasks import dummy_task

dummy_task.apply_async(countdown=4)

Expected Behavior

The message should be retried twice and then discarded.

Actual Behavior

The message is dropped by rabbitmq, because it detects a cycle. That way it is stuck in the RabbitMQ queue as dead-lettered:

Image

This bug is an interplay between native delayed delivery, and an ETA/Countdown task that is retried. In the example given above, celery puts the task in delayed_delivery_2 for the 4s initial delay. Then it is executed by the worker and (because it fails and is retried) is sent to RabbitMQ again with all the previous headers. When it reaches the queue delayed_delivery_2 again during the retry, RabbitMQ will detect a cycle and drop it.

We investigated this quite a bit and this is probably, because celery copies all the headers when retrying a task. This includes the "X-Death" header, which apparently RabbitMQ uses to detect cycles. This header should probably stripped when retrying tasks in celery.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions