Checklist
Mandatory Debugging Information
Optional Debugging Information
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Environment & Settings
Celery version: 5.5.3
celery report Output:
software -> celery:5.5.3 (immunity) kombu:5.6.0b2 py:3.12.8
billiard:4.2.1 py-amqp:5.3.1
platform -> system:Linux arch:64bit, ELF
kernel version:6.8.0-51-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:disabled
deprecated_settings: None
broker_url: 'amqp://user:********@localhost:5672/my_host'
task_default_exchange_type: 'topic'
task_queues: [<unbound Queue celery-quorum -> <unbound Exchange celery-quorum(topic)> -> celery-quorum>]
task_default_queue: 'celery-quorum'
broker_transport_options:
'confirm_publish': True}
Steps to Reproduce
Required Dependencies
- Minimal Python Version: N/A or Unknown
- Minimal Celery Version: N/A or Unknown
- Minimal Kombu Version: N/A or Unknown
- Minimal Broker Version: N/A or Unknown
- Minimal Result Backend Version: N/A or Unknown
- Minimal OS and/or Kernel Version: N/A or Unknown
- Minimal Broker Client Version: RabbitMQ 3.13.7 and 4.1.2
- Minimal Result Backend Client Version: N/A or Unknown
Python Packages
pip freeze Output:
amqp==5.3.1
billiard==4.2.1
boto3==1.40.13
botocore==1.40.13
celery @ git+https://github.com/celery/celery@33eb14852310996b1909c8388cd319809d6c8626
certifi==2025.8.3
charset-normalizer==3.4.3
click==8.2.1
click-didyoumean==0.3.1
click-plugins==1.1.1.2
click-repl==0.3.0
debugpy==1.8.16
docker==7.1.0
idna==3.10
iniconfig==2.1.0
jmespath==1.0.1
kombu==5.6.0b2
packaging==25.0
pluggy==1.6.0
prompt-toolkit==3.0.51
psutil==7.0.0
pygments==2.19.2
pytest==8.4.1
pytest-celery==1.2.1
pytest-docker-tools==3.1.9
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
python-memcached==1.62
redis==6.4.0
requests==2.32.5
s3transfer==0.13.1
setuptools==80.9.0
six==1.17.0
tenacity==9.1.2
tzdata==2025.2
urllib3==1.26.20
vine==5.1.0
wcwidth==0.2.13
Other Dependencies
Details
N/A
Minimally Reproducible Test Case
Details
# celeryconfig.py
RABBITMQ_USERNAME = os.getenv('RABBITMQ_USERENAME', 'guest')
RABBITMQ_PASSWORD = os.getenv('RABBITMQ_PASSWORD', 'guest')
app = Celery('celery_test')
app.conf.update(
broker_url=f'pyamqp://{RABBITMQ_USERNAME}:{RABBITMQ_PASSWORD}@localhost:5672/my_vhost',
task_default_exchange_type='topic',
task_queues=[
Queue(
'celery-quorum',
exchange=Exchange('celery-quorum', type='topic'),
routing_key='celery-quorum',
queue_arguments={'x-queue-type': 'quorum'}
),
],
task_default_queue='celery-quorum',
broker_transport_options={"confirm_publish": True},
)
app.autodiscover_tasks(['tasks'])
# tasks.py
@app.task(autoretry_for=(ValueError,), retry_kwargs={"max_retries": 2},default_retry_delay=15)
def dummy_task():
print("Dummy task is running")
raise ValueError("Dummy task is broken")
# client.py
from tasks import dummy_task
dummy_task.apply_async(countdown=4)
Expected Behavior
The message should be retried twice and then discarded.
Actual Behavior
The message is dropped by rabbitmq, because it detects a cycle. That way it is stuck in the RabbitMQ queue as dead-lettered:
This bug is an interplay between native delayed delivery, and an ETA/Countdown task that is retried. In the example given above, celery puts the task in delayed_delivery_2 for the 4s initial delay. Then it is executed by the worker and (because it fails and is retried) is sent to RabbitMQ again with all the previous headers. When it reaches the queue delayed_delivery_2 again during the retry, RabbitMQ will detect a cycle and drop it.
We investigated this quite a bit and this is probably, because celery copies all the headers when retrying a task. This includes the "X-Death" header, which apparently RabbitMQ uses to detect cycles. This header should probably stripped when retrying tasks in celery.
Checklist
mainbranch of Celery.contribution guide
on reporting bugs.
for similar or identical bug reports.
for existing proposed fixes.
to find out if the bug was already fixed in the main branch.
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
celery -A proj reportin the issue.(if you are not able to do this, then at least specify the Celery
version affected).
mainbranch of Celery.pip freezein the issue.to reproduce this bug.
Optional Debugging Information
and/or implementation.
result backend.
broker and/or result backend.
ETA/Countdown & rate limits disabled.
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Environment & Settings
Celery version: 5.5.3
celery reportOutput:Steps to Reproduce
Required Dependencies
Python Packages
pip freezeOutput:Other Dependencies
Details
N/A
Minimally Reproducible Test Case
Details
Expected Behavior
The message should be retried twice and then discarded.
Actual Behavior
The message is dropped by rabbitmq, because it detects a cycle. That way it is stuck in the RabbitMQ queue as dead-lettered:
This bug is an interplay between native delayed delivery, and an ETA/Countdown task that is retried. In the example given above, celery puts the task in
delayed_delivery_2for the 4s initial delay. Then it is executed by the worker and (because it fails and is retried) is sent to RabbitMQ again with all the previous headers. When it reaches the queuedelayed_delivery_2again during the retry, RabbitMQ will detect a cycle and drop it.We investigated this quite a bit and this is probably, because celery copies all the headers when retrying a task. This includes the "X-Death" header, which apparently RabbitMQ uses to detect cycles. This header should probably stripped when retrying tasks in celery.