Test: a real worker failing at each stage, for a task and a chord (#10724, #10725) - #10726

…c1 (#160)

## TLDR

Today the Celery worker self-test pins two outcomes that Celery has
since fixed. When the pool child is lost after SUCCESS is stored, or the
task's `on_success` hook raises, the test still expects the error
callback to fire beside the link. Celery fixed both on `main`
(celery/celery#10752 and #10759, for celery/celery#10724), after
5.7.0rc1. With the lock still on 5.6.3, CI is green. The first lock bump
to a release with the fixes would turn this suite red, though nothing in
the harness was wrong.

This PR keeps those two rows only for Celery releases up to 5.7.0rc1. So
the table holds on both sides of the fix, and a lock bump no longer
breaks it. The rows that still diverge are now linked to their open
issues: `died at mark_as_done` to celery/celery#10949 and the refused
link to celery/celery#10725.

| Celery | `died at task_postrun` | `the task's on_success hook raised`
|
| --- | --- | --- |
| ≤ 5.7.0rc1 (the lock: 5.6.3) | pinned: SUCCESS, `failed:
WorkerLostError` + `sent` | pinned: SUCCESS, `failed: ReceiverFailed` +
`sent` |
| later releases | normal operation: SUCCESS, `sent` | normal operation:
SUCCESS, `sent` |

The gap text drops "after the result is stored". It now reads "a failure
after the link is published, or a refused link, still runs the failure
path", which is true on both sides.

## Why a version gate

Celery's `main` between those merges and its next release still reports
`5.7.0rc1`. Detecting #10759 would mean reading Celery's source, and the
suite runs the locked release anyway. So the gate is
`Version(celery.__version__) > Version("5.7.0rc1")`, with that caveat
written beside it.

## Tested

In WSL, Python 3.12, Redis 7.4.11 (`package/tests/celery_worker`, `-p
no:django -rxX`):

| Celery | Gate | Result |
| --- | --- | --- |
| 5.6.3 (the lock) | off, as computed | 15 passed, 2 xfailed |
| `main` at e3cc7cc53 (has both fixes), gate forced on | on | 15 passed,
2 xfailed |
| `main` at e3cc7cc53, gate forced off | off | fails, naming `died at
task_postrun` as moved, as the table should |

- `ruff check` and `ruff format --check`: clean.
- `pyrefly check`: no errors in the changed file. The environment lacked
the extras and demos behind the repository's other errors.

Found while updating celery/celery#10726, which runs the same
integration against Celery's `main`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>

…ate history= (#161)

## TLDR

Today `celery_worker.worker_contract()` binds exactly one history, so an
adopter checking a task and a chord has to `model_copy` the second
history onto `process_handoffs` and `handoff_gaps` by hand.
celery/celery#10726 does exactly that. This PR makes `worker_contract`
take its histories the way `ObligationContract` itself does:
`process_handoffs=` with `handoff_gaps=` keyed by history name, the same
field names with the same meaning, so nothing is translated between two
vocabularies. The one-history form, `history=` with `gap=`, is
deprecated: it still binds, and emits a `DeprecationWarning` naming the
replacement. A mix of the two forms is refused in words.

The reference self-test now binds a chord of two `send_message` tasks
next to the single task. On the locked Celery 5.6.3, that chord
reproduces celery/celery#10724's chord case on a real worker: a header
lost after its SUCCESS was stored, or whose `on_success` raised, fails
the whole chord.

Builds on #160 (merged), which gates the fixed rows on the Celery
release.

```text
BEFORE                                            AFTER
worker_contract(history=TASK, gap=...)            worker_contract(process_handoffs=(TASK,), handoff_gaps={TASK.name: ...})
                                                  (history=/gap= still work and warn: DeprecationWarning)
a second history: model_copy(update={             worker_contract(process_handoffs=(TASK, CHORD),
  "process_handoffs": (..., CHORD),                   handoff_gaps={TASK.name: ..., CHORD.name: ...})
  "handoff_gaps": {..., CHORD.name: ...}})
```

## What the chord history pins

The faults hit the first execution of a header task. The refused publish
is the last header publishing the body.

| History | Celery ≤ 5.7.0rc1 (lock 5.6.3) | Later releases (verified on
Celery `main`) |
| --- | --- | --- |
| normal operation | SUCCESS, 2 sent, `sent 2 messages` | same |
| died at `task_prerun` | FAILURE, 1 sent, `failed: ChordError` | same |
| died at `mark_as_done` | FAILURE, 2 sent, `failed: ChordError`
(celery/celery#10949) | same |
| died at `task_postrun` | FAILURE, 2 sent, `failed: ChordError`
(celery/celery#10724) | normal operation |
| `on_success` raised | FAILURE, 2 sent, `failed: ChordError`
(celery/celery#10724) | normal operation |
| broker refused the body | FAILURE, 2 sent, `failed: ChordError`
([#10725
comment](celery/celery#10725 (comment)))
| same |

## Refused bindings

`worker_contract` asserts, naming the contract:
- not both the deprecated `history=` and `process_handoffs=` or
`handoff_gaps=`;
- no `gap=` without `history=`;
- at least one history.

Two histories sharing a name, and a `handoff_gaps` key naming no bound
history, are refused by `ObligationContract`'s own design checks
(`ObligationContractDesignError`), so `worker_contract` doesn't repeat
them; a test pins that they apply to several histories.

## Reference test and docs

- Both histories are built by one `reference_history(name, send,
findings)` helper instead of two near-identical `worker_history` calls.
- The rows #10752/#10759 remove are added through one
`before_the_10724_fixes(rows)` helper beside the gate, which now states
its retirement condition: delete it and those rows once `uv.lock`'s
Celery is newer than 5.7.0rc1.
- `package/docs/integrations.md` describes the binding beside the
integrations table, with an example and the deprecation.

## Tested

WSL, Python 3.12, Redis 7.4.11, `package/tests/celery_worker -p
no:django -rxX`:

| Celery | Result |
| --- | --- |
| 5.6.3 (the lock) | 15 passed, 3 xfailed |
| `main` at e3cc7cc53, release gate forced on | 15 passed, 3 xfailed |

- `test_celery_worker_contract_arguments.py`: 8 passed (the binding, the
deprecated form binding with its warning, 4 mixed-form refusals, 2 core
design refusals).
- After these changes, `package/tests/celery_worker` on Celery 5.6.3
with real workers: 23 passed, 3 xfailed. It needs no worker, so it also
runs on Windows.
- `ruff check package` and `ruff format --check package`: clean.
- `pyrefly check`: no errors in `celery_worker` files.

The 5.6.3 run shows 7 `DeprecationWarning`s from Celery's own Redis
backend (`setex`) on the chord path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>

…ions (#186)

## TLDR

Contracts could claim regression coverage while observing too little,
declaration-only cases appeared as passing tests, and Redis examples
could erase developer databases. This PR adds reusable regression
controls, honest execution reporting, disposable Redis support, and
agent guidance for building contracts maintainers can trust. Celery
supplies the real-system regression example.

The changes strengthen two existing invariants: a passing behavioral
count represents executed checks, and a claimed regression must be
visible to the contract's unchanged observation. The Redis helper
establishes explicit ownership of destructive test resources.

```text
BEFORE
agent -> inject fault -> aggregate XFAIL / inflated pass count -> coverage claim
test  -> default localhost Redis -> FLUSHDB

AFTER
agent -> passing baseline -> targeted regression -> specific changed observation
      -> separate proofs, declarations and per-stage diagnostics -> scoped claim
test  -> owned disposable Redis -> isolated worker databases -> teardown
```

## Changes

- Add framework-independent `assert_detects_regression`. It requires a
passing baseline, the intended assertion under a regression context, and
a passing check after restoration; setup, cleanup and unrelated failures
do not count as detection.
- With `--due-work-summary`, report no-op exclusions as `DECLARED` and
exclude them from the terminal proof count. JSON coverage always
excludes declarations from passing proofs; JUnit retains their reasons.
Declaration calls complete normally, preserving consumers that reject
skips and existing pytest outcome counts. Executable assessment probes
remain checks. Setup/teardown failures remain visible, including across
report shards.
- Attach per-stage history comparisons to pytest summaries and JUnit,
including under xdist. Stage rows are diagnostics, not additional tests;
observed product values are not serialized into this metadata.
- Add a disposable Redis context/command runner. Destructive self-tests
require this runner or explicit dedicated-server authorization. CI uses
owned processes and clears inherited Celery overrides that would defeat
worker isolation.
- Share the owned Redis subprocess environment between the runner and
reversion check, and extract their reference observer into a support
module so regression verification does not import a generated test
suite.
- Record chord header errors in the reference observer, update the
observed findings, and correct stale issue links.
- Add a pinned upstream reversion check to CI and update both portable
skills, the repository skill, adoption documentation and disclosure
templates. Guidance now requires sensitivity evidence, resource
ownership, accurate counts, useful standalone reproducers and restrained
upstream communication.

## Before/after evidence and policy

| Scenario / metric | Before | After | Evidence and meaning |
| --- | --- | --- | --- |
| Celery fixes detected by the chord observer, higher is better | 0 of 2
reverted fixes | 2 of 2 | Real workers and Redis, same pinned source and
targeted faults |
| Derivation example with `--due-work-summary` | 17 | 8 checks + 9
declarations | Serial and xdist; ordinary pytest counting remains
compatible, JSON separates declarations in either mode |
| Redis reset policy | Implicit localhost databases | Owned subprocesses
by default; dedicated-server opt-in for direct self-test runs | Another
server's sentinel survives; cleanup and failed startup tested |
| Fault-stage evidence | Aggregate case outcome | Individual comparisons
alongside the unchanged aggregate verdict | XFAIL, xdist and JUnit
integration tests |

## Validation

- Core plus application-core integration tests after review: **1,534
passed, 9 optional skips**. A consumer hook reproduces Due's strict skip
rejection; the fix passes in both ordinary and declaration-aware
reporting modes. Five real Redis lifecycle controls pass.
- Redis/RQ/Celery and application integrations, four workers: **444
passed, 10 known XFAILs, 85 declarations** in CI with exploration
enabled. Executed-adoption and replay validators pass.
- `scripts/check_celery_regressions.py`: Celery pinned at
`e3cc7cc532a9a0ef0be575a66a03cfff52ef23be`; reverse #10752 and #10759
independently. The old observer passes despite each regression. The
improved observer passes the baseline, rejects each regression, and
verifies the restored baseline.
- 20 CI routing tests pass. Lint/format, changed-file Pyrefly checks,
local documentation links and all three edited skills validate. An
isolated core-only install imports 98 core modules without a framework.
- Local runtime: Python 3.12; the reference integration suite uses
locked Celery 5.6.3. The reversion experiment uses the pinned upstream
checkout, not that installed release.
- CI is green on review-fix commit `7bb7d7c`, including the aggregate
Validation gate, Python 3.11–3.14, framework integrations, demos,
Saleor, typing, lint, docs and Conventional Commits. The unchanged
pinned Due checkout passes **959 library tests** and **968 conformance
tests with 2 expected failures**, plus all adoption checks. The Redis
tutorial passes **47 tests with 3 expected failures**. [CI
run](https://github.com/gigaverse-app/pytest-obligation/actions/runs/40436061425)
- No repository PR template was found.

## Limits and adoption

The regression helper proves sensitivity to the supplied bad behavior;
it cannot establish observer completeness automatically. This PR does
**not** claim to guard Redis chord deduplication (#10919), which needs a
distinct duplicate-header control. The Redis process helper requires
POSIX and a local `redis-server`; it does not sandbox test code that
ignores the supplied endpoints.

Use `python -m pytest_obligation.integrations.redis_server -- pytest
...` for destructive Redis self-tests. Existing application bindings and
default pytest outcomes remain compatible. Use `--due-work-summary` for
separate declaration counts or the JSON coverage report for behavioral
evidence; default pytest pass totals include completed declarations.
Full-project typing was not validated locally because unrelated
demo/framework dependencies were not installed; changed files pass
typing and CI runs the repository configuration.

Motivation: [Celery
review](celery/celery#10726 (comment)).
No upstream issue, comment, or PR was modified.