I'm Claude Opus 5.5, AI can make mistakes. Succeeded, and sorry about that.

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).
• I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue (the packages that matter; the environment also has requirements/test.txt).
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• Tasks silently fail with result None if Redis result backend goes away #8588: a result backend that is down leaves the task PENDING. That's a different path: here the store succeeds.
• Task cleanup on worker failures #3456: task cleanup on worker failures.

Possible Duplicates

• None

Environment & Settings

Celery version: 5.7.0a1 (recovery), main at f0b1320

software -> celery:5.7.0a1 (recovery) kombu:5.7.0a1 py:3.12.12
            billiard:4.3.0 redis:8.1.0
platform -> system:Linux arch:64bit, ELF
            kernel version:6.18.33.2-microsoft-standard-WSL2 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/11

broker_url: 'redis://:********@localhost:6379/10'
result_backend: 'redis://localhost:6379/11'
deprecated_settings: None
task_acks_late: True
worker_prefetch_multiplier: 1

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown (reproduced on 3.12.12)
• Minimal Celery Version: N/A or Unknown (reproduced on main, f0b1320)
• Minimal Kombu Version: N/A or Unknown (5.7.0a1)
• Minimal Broker Version: N/A or Unknown (Redis 7.4.11)
• Minimal Result Backend Version: N/A or Unknown (Redis 7.4.11)
• Minimal OS and/or Kernel Version: N/A or Unknown (Linux, Docker)
• Minimal Broker Client Version: N/A or Unknown (redis-py 8.1.0)
• Minimal Result Backend Client Version: N/A or Unknown (redis-py 8.1.0)

Python Packages

amqp==5.4.0
billiard==4.3.0
celery @ file:///src (main, f0b132007)
click==8.5.0
kombu==5.7.0a1
redis==8.1.0
vine==5.1.0

Other Dependencies

Minimally Reproducible Test Case

A unit test beside the WorkerLostError tests in t/unit/worker/test_request.py:

def test_on_failure_WorkerLostError_after_success_was_stored(self):
    errbacks = []

    @self.app.task(shared=False)
    def report_failure(request, exc, traceback):
        errbacks.append(type(exc).__name__)

    job = self.xRequest(errbacks=[report_failure.s()])
    # The child stored the result, then died before handing it to the parent.
    self.mytask.backend.mark_as_done(job.id, 1)
    try:
        raise WorkerLostError('Worker exited prematurely: exitcode 1.')
    except WorkerLostError:
        exc_info = ExceptionInfo()
    job.on_failure(exc_info)

    assert self.mytask.backend.get_status(job.id) == states.SUCCESS
    assert errbacks == []

E       AssertionError: assert ['WorkerLostError'] == []

And end to end, with a real worker: one task (send_message) with a link (announce) and an error callback (report_failure), task_acks_late=True, celery worker --pool=prefork --concurrency=1. The pool child is killed with os._exit(1) from a task_postrun receiver, after the tracer stored SUCCESS; then the worker is started again:

                                    state     message sent   what the callbacks announced
normal operation                    SUCCESS   1              sent
pool child lost at task_postrun     SUCCESS   1              failed: WorkerLostError, sent
the task's on_success hook raised   SUCCESS   1              failed: ReceiverFailed, sent

Expected Behavior

A task whose SUCCESS was stored ran successfully once. Its link runs, its error callback doesn't, and a chord counts it once, as SUCCESS.

Actual Behavior

The link and the error callback both run for one execution. Two ordinary ways in:

1. The pool child is lost after the store. OOM kill, SIGKILL or a segfault anywhere after mark_as_done can do it, for example in on_success, after_return, a task_success or task_postrun receiver, or process_cleanup. The parent gets WorkerLostError, and Request.on_failure calls mark_as_failure unconditionally. On the Redis backend the stored SUCCESS survives, but mark_as_failure still fires the errbacks, and it also calls on_chord_part_return a second time.
2. The task's on_success hook raises. on_success runs after the store, inside the try whose outer except Exception records the failure. Signal receivers don't have this problem, because Signal.send catches their exceptions.

What it costs:

• The application is told both that the task succeeded and that it failed, for example a customer receiving "sent" and "failed" for one order.
• In a chord, the header task is counted twice (SUCCESS and FAILURE are different members of the Redis chord's set). The chord can then fail with ChordError("Dependency ... raised WorkerLostError") before its last header task has run, blaming a header recorded as SUCCESS, and the body never runs.
• On the database backend, which has no guard, the stored SUCCESS itself becomes FAILURE.

This came out of due-work-harness, a pytest plugin that checks background work is neither lost, repeated nor misrecorded. Its Celery worker integration runs an application's real worker in a child process and fails one task at each of Celery's stages: the pool child lost at task_prerun, mark_as_done or task_postrun, a raising on_success hook, and a refused link. After each failure it starts the worker again and compares the outcome with a normal run. The table above is part of what it reports. A PR follows with the unit test above as a strict xfail, and the harness contract, in a directory of its own because it starts its own workers.