Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).
• I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue (the packages that matter; the environment also has requirements/test.txt).
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• Rest of chain is lost after a task is retried chained to pre-work (another task) #6631: the rest of a chain is lost after a retried task. A different trigger, the same outcome for the chain.
• Task cleanup on worker failures #3456: task cleanup on worker failures.

Possible Duplicates

• None

Environment & Settings

Celery version: 5.7.0a1 (recovery), main at f0b1320

software -> celery:5.7.0a1 (recovery) kombu:5.7.0a1 py:3.12.12
            billiard:4.3.0 redis:8.1.0
platform -> system:Linux arch:64bit, ELF
            kernel version:6.18.33.2-microsoft-standard-WSL2 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/11

broker_url: 'redis://:********@localhost:6379/10'
result_backend: 'redis://localhost:6379/11'
deprecated_settings: None
task_acks_late: True
worker_prefetch_multiplier: 1

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown (reproduced on 3.12.12)
• Minimal Celery Version: N/A or Unknown (reproduced on main, f0b1320)
• Minimal Kombu Version: N/A or Unknown (5.7.0a1)
• Minimal Broker Version: N/A or Unknown (Redis 7.4.11)
• Minimal Result Backend Version: N/A or Unknown (Redis 7.4.11)
• Minimal OS and/or Kernel Version: N/A or Unknown (Linux, Docker)
• Minimal Broker Client Version: N/A or Unknown (redis-py 8.1.0)
• Minimal Result Backend Client Version: N/A or Unknown (redis-py 8.1.0)

Python Packages

amqp==5.4.0
billiard==4.3.0
celery @ file:///src (main, f0b132007)
click==8.5.0
kombu==5.7.0a1
redis==8.1.0
vine==5.1.0

Other Dependencies

Minimally Reproducible Test Case

With a real worker: one task (send_message) with a link (announce) and an error callback (report_failure), task_acks_late=True, celery worker --pool=prefork --concurrency=1, Redis as broker and backend. In the worker, the first Signature.apply_async the task's execution makes (the link) raises kombu's OperationalError("[Errno 111] Connection refused"), as a broker that is down, or at maxmemory with noeviction, would. Then the worker is started again:

                                      state     message sent   what the callbacks announced   broker
normal operation                      SUCCESS   1              sent                           queue 0, unacked 0
the broker refused the task's link    FAILURE   1              failed: OperationalError       queue 0, unacked 0

The contract that produces this, t/due_work/test_worker_contract.py, is in the PR that follows.

Expected Behavior

The task's body ran and its effect happened. Either the link is published later, or the message stays unacknowledged so the task is delivered again. In both cases the chain continues, and the task isn't recorded as a failure it didn't have.

Actual Behavior

The side effect happened, the result says FAILURE, the error callback fires, and the message is acknowledged even with acks_late. So the link is never sent, and a chain stops there for good.

_dispatch_callbacks_and_chain runs before mark_as_done, and only EncodeError is handled around it (L628). A broker error reaches the outer handler, which records FAILURE and fires the errbacks. Request.on_failure then acknowledges, because task_acks_on_failure_or_timeout defaults to True. Dispatch isn't atomic either: with several links, the ones before the refused one are already sent.

It also happens with a broker that really refuses the write, not only an injected error: with the Redis broker full (maxmemory 1, maxmemory-policy noeviction) when the link is published, Redis rejects the publish with OutOfMemoryError, kombu raises OperationalError, and the task ends FAILURE with its error callback fired, the message acknowledged (queue 0, unacked 0) and the link never sent, even after the worker restarts. A broker that only pauses writes (CLIENT PAUSE ... WRITE) does not trigger it: the publish waits and succeeds. The script is in this comment on #10726.

Aside: how this was found. This came from pytest-obligation, an open-source pytest plugin for background work that must not be lost, run twice or recorded wrongly. Given one task and how to see what it did, it generates the failure cases on a real celery worker: it kills the pool child at each of Celery's stages, makes a hook raise, refuses a publish, restarts the worker, and compares every outcome with a normal run. In Celery it produced #10724 (fixed by #10752 and #10759), #10725 and #10954. Each stays pinned as a strict xfail, so a fix shows up as a change, and so does a regression; when the two #10724 fixes landed, the unchanged contract flagged exactly those two cases. It runs nightly against Celery's main on gigaverse-app/celery, and #10726 shows what adopting it in Celery's own CI would look like.

Written with Claude Code at Aviad Rozenhek's direction; every result above was produced by running the code shown.