Skip to content

feat(celery): bind worker histories with process_handoffs= and deprecate history= - #161

Merged
aviadr1 merged 9 commits into
mainfrom
claude/celery-worker-histories
Oct 25, 2026
Merged

aviadr1 merged 9 commits into
mainfrom
claude/celery-worker-histories

Conversation

@aviadr1

@aviadr1 aviadr1 commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Today celery_worker.worker_contract() binds exactly one history, so an adopter checking a task and a chord has to model_copy the second history onto process_handoffs and handoff_gaps by hand. celery/celery#10726 does exactly that. This PR makes worker_contract take its histories the way ObligationContract itself does: process_handoffs= with handoff_gaps= keyed by history name, the same field names with the same meaning, so nothing is translated between two vocabularies. The one-history form, history= with gap=, is deprecated: it still binds, and emits a DeprecationWarning naming the replacement. A mix of the two forms is refused in words.

The reference self-test now binds a chord of two send_message tasks next to the single task. On the locked Celery 5.6.3, that chord reproduces celery/celery#10724's chord case on a real worker: a header lost after its SUCCESS was stored, or whose on_success raised, fails the whole chord.

Builds on #160 (merged), which gates the fixed rows on the Celery release.

BEFORE                                            AFTER
worker_contract(history=TASK, gap=...)            worker_contract(process_handoffs=(TASK,), handoff_gaps={TASK.name: ...})
                                                  (history=/gap= still work and warn: DeprecationWarning)
a second history: model_copy(update={             worker_contract(process_handoffs=(TASK, CHORD),
  "process_handoffs": (..., CHORD),                   handoff_gaps={TASK.name: ..., CHORD.name: ...})
  "handoff_gaps": {..., CHORD.name: ...}})

What the chord history pins

The faults hit the first execution of a header task. The refused publish is the last header publishing the body.

History Celery ≤ 5.7.0rc1 (lock 5.6.3) Later releases (verified on Celery main)
normal operation SUCCESS, 2 sent, sent 2 messages same
died at task_prerun FAILURE, 1 sent, failed: ChordError same
died at mark_as_done FAILURE, 2 sent, failed: ChordError (celery/celery#10949) same
died at task_postrun FAILURE, 2 sent, failed: ChordError (celery/celery#10724) normal operation
on_success raised FAILURE, 2 sent, failed: ChordError (celery/celery#10724) normal operation
broker refused the body FAILURE, 2 sent, failed: ChordError (#10725 comment) same

Refused bindings

worker_contract asserts, naming the contract:

  • not both the deprecated history= and process_handoffs= or handoff_gaps=;
  • no gap= without history=;
  • at least one history.

Two histories sharing a name, and a handoff_gaps key naming no bound history, are refused by ObligationContract's own design checks (ObligationContractDesignError), so worker_contract doesn't repeat them; a test pins that they apply to several histories.

Reference test and docs

  • Both histories are built by one reference_history(name, send, findings) helper instead of two near-identical worker_history calls.
  • The rows #10752/#10759 remove are added through one before_the_10724_fixes(rows) helper beside the gate, which now states its retirement condition: delete it and those rows once uv.lock's Celery is newer than 5.7.0rc1.
  • package/docs/integrations.md describes the binding beside the integrations table, with an example and the deprecation.

Tested

WSL, Python 3.12, Redis 7.4.11, package/tests/celery_worker -p no:django -rxX:

Celery Result
5.6.3 (the lock) 15 passed, 3 xfailed
main at e3cc7cc53, release gate forced on 15 passed, 3 xfailed
  • test_celery_worker_contract_arguments.py: 8 passed (the binding, the deprecated form binding with its warning, 4 mixed-form refusals, 2 core design refusals).
  • After these changes, package/tests/celery_worker on Celery 5.6.3 with real workers: 23 passed, 3 xfailed. It needs no worker, so it also runs on Windows.
  • ruff check package and ruff format --check package: clean.
  • pyrefly check: no errors in celery_worker files.

The 5.6.3 run shows 7 DeprecationWarnings from Celery's own Redis backend (setex) on the chord path.

🤖 Generated with Claude Code

aviadr1 and others added 5 commits October 25, 2026 10:12
worker_contract takes histories= with handoff_gaps= keyed by history name,
as RQ's does; history= and gap= remain the single-history form. The
reference test adds a chord of two send_message tasks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@read-the-docs-community

read-the-docs-community Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 pytest-obligation | 🛠️ Build #35389657 | 📁 Comparing 365cea1 against latest (7ddb409)

  🔍 Preview build  

1 file changed
± package/docs/integrations/index.html

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

pytest-obligation: what passed, what is missing, what to fix

3 applications196 checks passed0 failures0 blocked checks0 coverage gaps0 rejected bindingsskipped tests: unknown
applications · checks passed · failures · blocked checks · coverage gaps · rejected bindings · skipped tests

2.5s elapsed · 2.2s summed check time

Application inventory and skipped tests

3 applications with results · 0 applications not run
Checks skipped: unknown.

Application inventory was not supplied; uncollected suites cannot be listed.

example catalog     194 passed0 failed0 blocked0 gaps0 rejected

View contract setup · examples/adopter/tests/test_catalog.py:7 · tested revision

Test time: 2.2s summed check time

⚪ Deliberate exclusions
  • Unfinished work gets recovered (A) · Not claimed by application. Application reason: This small example exposes reconcile explicitly; it ships no periodic scheduler.
  • Worker ownership is bounded (B) · Not applicable (application decision). Application reason: A single serial executor runs the example; no lease changes hands.
  • Uncertain outcomes are handled (C) · Not claimed by application. Application reason: Remote assignments are repeatable; reconciliation compares actual remote values.
  • Needed records are retained (D) · Not applicable (application decision). Application reason: The example has no pruning command; durable owners are retained.
  • Results eventually settle correctly (E) / STALE_SNAPSHOTS · Not applicable (application decision). Application reason: No separate snapshot merge API; in-flight revisions exercise the worker.
  • Results eventually settle correctly (E) / MONOTONIC_RESULTS · Not applicable (application decision). Application reason: The product accepts reversible colors; receipts use evidence confluence.
  • Owed work is discovered from stored facts (F) · Not applicable (application decision). Application reason: Admission creates explicit obligation rows; recovery does not derive missing rows.

These are scope decisions, not demonstrated defects. Review their reasons when the application changes.


Hypothesis · 3/3 scenario checks passed

Generated schedule exploration uses the application's scenario bindings.

  • Application scenario catalog retirement: Passed
  • Application scenario catalog revisions: Passed
  • Application scenario delivery receipts: Passed

These counts are scenario checks, not generated examples. Example counts, shrinking statistics and replay histories are not recorded in this report; inspect the pytest log for a failing history.


🟢 5 supported guarantees
  • Results eventually settle correctly
  • Execution respects its gate
  • Repeating work is harmless
  • Intent and work are saved together
  • Retries have a limit

Suite (application code): adopter.tests.test_catalog.TestCatalog

Noticeboard.publish…1 passed0 failed0 blocked0 gaps0 rejected

View suite · examples/adopter/tests/test_notification_exemption.py:54 · tested revision

Test time: 0.0s summed check time

Application: Noticeboard.publish exemption

Scope: Handoff exemption. Application reason: Publishing commits the complete notice; the optional alert neither finishes nor changes that notice.


Hypothesis · Not reported

This report has no generated exploration evidence.


🟢 1 supported guarantees
  • Handoff exemption
    • Application reason: Publishing commits the complete notice; the optional alert neither finishes nor changes that notice.

Suite (application code): adopter.tests.test_notification_exemption.TestOptionalAlert

rename_order exempt…1 passed0 failed0 blocked0 gaps0 rejected

View suite · examples/adopter/tests/test_summaries_exemption.py:12 · tested revision

Test time: 0.0s summed check time

Application: rename_order exemption

Scope: Handoff exemption. Application reason: the read path rebuilds a summary that disagrees with its order, so a lost eviction costs nothing


Hypothesis · Not reported

This report has no generated exploration evidence.


🟢 1 supported guarantees
  • Handoff exemption
    • Application reason: the read path rebuilds a summary that disagrees with its order, so a lost eviction costs nothing

Suite (application code): adopter.tests.test_summaries_exemption.TestSummaryEvictionExemption


What do these numbers mean?
  • Failures: an executed assertion failed unexpectedly or was an application-declared expected failure. Expected failures still permit successful CI; the visible breakdown distinguishes them.
  • Rejected bindings: repair the connection to production code; these checks provide no supporting evidence.
  • Blocked checks: checks could not establish a result, including missing execution, setup/cleanup errors, binding-debt or unclassified expected failures, and unexpected passes requiring declaration review.
  • Coverage gaps: guarantees not assessed, missing scenarios, or with declared gaps. Each affected guarantee counts once. Deliberate exclusions are shown separately.

Pass/fail/blocked/rejected counts partition unique checks. Gaps count guarantees, not extra tests. Declined and not-applicable declarations remain separate and do not establish passing coverage. Graphics show zero issues in green; missing reports never show green zeroes.

Elapsed time covers the pytest session. Summed check time includes setup, execution and cleanup; parallel work overlaps.
Application totals include discovered declarations and reported suites. Skipped checks count known deselections and explicit pytest skips; incomplete selected checks are not skips. Uncollected applications have unknown check counts. Zero recorded outcomes do not certify passing coverage.

Application names, quoted reasons and diagnostic values come from application code. Guarantee labels come from pytest-obligation. Results cover collected suites only; passing checks do not rate every test's quality.

Test step (GitHub): success
Source: PR head 365cea14bbd38a5a5efa885b07577504c80091b0 · workflow commit 8b9db5931223bce69d9d53d03137d03630f68673 · run 40179895071 · attempt 1

Workflow run · Report guide

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

CI-esta pull request report

merge-ready: 4m 20s selected: 16 / 16 lanes executed: 22 jobs reused: unavailable runner: 48 min est. cost: $0.00

✓ Required checks passed CI · 365cea1 · attempt 1 · complete snapshot

Runners: ubuntu-latest


Selection and execution
 16 / 16 lanes selected · 22 jobs executed · unavailable reused

Selected is the plan; executed is observed; reused requires a receipt. Units can differ.

When executed is shown in jobs, it counts started jobs in this attempt and includes setup, checks and reporting. Measured shards count child processes, including failed ones; neither count asserts test success.

Test lanes
 16 / 16 lanes selected
 Plan before evidence reuse

🧪 Test lanes · by pytest-selective

🟧🟧🟧🟧🟧🟧🟧🟧🟧🟧🟧🟧🟧🟧🟧🟧 · 5 changed files. These are planning decisions; receipt reuse and execution outcomes are reported separately.

Every lane was selected: dependency closure: 3 test files.

Why these lanes?
Lane Because
core selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
minimum_models selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
exploration selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
django selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
django_psycopg2 selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
redis selected affected test package/tests/celery_worker/test_celery_worker_contract.py
affected test package/tests/celery_worker/test_celery_worker_contract_arguments.py
affected test package/tests/celery_worker/test_celery_worker_platform.py
+2 more
mongodb selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
aiokafka selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
prefect selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
demos selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
saleor selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
actions selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
due selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
typecheck selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
explicit input package/tests/celery_worker/app.py
explicit input package/tests/celery_worker/test_celery_worker_contract.py
+1 more
contracts selected explicit input package/src/pytest_obligation/integrations/celery_worker.py
explicit input package/tests/celery_worker/app.py
explicit input package/tests/celery_worker/test_celery_worker_contract.py
+1 more
wheel selected explicit input package/src/pytest_obligation/integrations/celery_worker.py

What else avoided work?
 unavailable reused · 13/13 environment cache hits

Reused validation evidence

Reuse evidence was not recorded. Skipped jobs alone do not prove reuse.

Environment caching

Job Observed outcome after locked sync Validation / sync
aiokafka hit 2s
django-psycopg2 hit 1s
django hit 1s
redis hit 1s
exploration hit 1s
core hit 1s
core hit 1s
minimum-models hit 1s
prefect hit 1s
mongodb hit 1s
core hit 1s
core hit 1s
lint hit 0s

Every recorded environment passed locked synchronization. A restored but unusable environment is recorded as repaired. Download caches are separate and are not included in these counts.

Cache time saved is unavailable without a comparable cold run.

Incremental test evidence is not inferred from skipped jobs or added to whole-lane reuse totals.


CI time and cost
 4m 20s merge-ready · 48 min runner · $0.00 est. cost

Runner minutes use per-job rounding; cost is an estimate.

Workflow Merge-ready First runner Longest job Runner time Cost
CI 4m 20s after 2s saleor · 3m 58s 48 min $0
Timeline and critical path
%%{init: {"theme": "base", "themeVariables": {"fontFamily": "-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif", "fontSize": "13px", "doneTaskBkgColor": "#3B82F6", "doneTaskBorderColor": "#2563EB", "critBkgColor": "#EA580C", "activeTaskBkgColor": "#2EA043", "activeTaskBorderColor": "#1A7F37", "critBorderColor": "#C2410C", "taskBkgColor": "#3B82F6", "taskBorderColor": "#2563EB", "taskTextColor": "#FFFFFF", "taskTextDarkColor": "#FFFFFF", "taskTextLightColor": "#FFFFFF", "taskTextOutsideColor": "#8B949E", "titleColor": "#8B949E", "sectionBkgColor": "rgba(234, 88, 12, 0.07)", "altSectionBkgColor": "rgba(139, 148, 158, 0.05)", "sectionBkgColor2": "rgba(234, 88, 12, 0.07)", "gridColor": "rgba(139, 148, 158, 0.3)", "todayLineColor": "rgba(0, 0, 0, 0)"}, "gantt": {"barHeight": 20, "barGap": 6, "topPadding": 44, "leftPadding": 230, "sectionFontSize": 13, "fontSize": 12}}}%%
gantt
    title 4m 20s to merge-ready · orange is what the merge waited on
    dateFormat YYYY-MM-DD HH:mm:ss
    axisFormat %M:%S
    todayMarker off
    section lint
    12s :done, j0, 2000-01-01 00:00:02, 2000-01-01 00:00:14
    section select-tests
    13s :crit, j1, 2000-01-01 00:00:02, 2000-01-01 00:00:15
    section actions
    38s :done, j2, 2000-01-01 00:00:17, 2000-01-01 00:00:55
    section aiokafka
    45s :done, j3, 2000-01-01 00:00:17, 2000-01-01 00:01:02
    section build
    16s :done, j4, 2000-01-01 00:00:17, 2000-01-01 00:00:33
    section ci-contracts
    26s :done, j5, 2000-01-01 00:00:17, 2000-01-01 00:00:43
    section core · 3.11
    2m 1s :done, j6, 2000-01-01 00:00:17, 2000-01-01 00:02:18
    section core · 3.12
    3m 21s :done, j7, 2000-01-01 00:00:17, 2000-01-01 00:03:38
    section demos
    3m 56s :done, j8, 2000-01-01 00:00:17, 2000-01-01 00:04:13
    section django
    1m 8s :done, j9, 2000-01-01 00:00:17, 2000-01-01 00:01:25
    section django-psycopg2
    1m 2s :done, j10, 2000-01-01 00:00:17, 2000-01-01 00:01:19
    section due
    2m 54s :done, j11, 2000-01-01 00:00:17, 2000-01-01 00:03:11
    section minimum-models
    2m 26s :done, j12, 2000-01-01 00:00:17, 2000-01-01 00:02:43
    section mongodb
    35s :done, j13, 2000-01-01 00:00:17, 2000-01-01 00:00:52
    section prefect
    1m 48s :done, j14, 2000-01-01 00:00:17, 2000-01-01 00:02:05
    section redis
    2m 31s :done, j15, 2000-01-01 00:00:17, 2000-01-01 00:02:48
    section saleor
    3m 58s :crit, j16, 2000-01-01 00:00:17, 2000-01-01 00:04:15
    section typecheck
    16s :done, j17, 2000-01-01 00:00:17, 2000-01-01 00:00:33
    section core · 3.13
    2m 18s :done, j18, 2000-01-01 00:00:18, 2000-01-01 00:02:36
    section core · 3.14
    2m 22s :done, j19, 2000-01-01 00:00:18, 2000-01-01 00:02:40
    section exploration
    2m 37s :done, j20, 2000-01-01 00:00:18, 2000-01-01 00:02:55
    section Validation gate
    3s :crit, j21, 2000-01-01 00:04:17, 2000-01-01 00:04:20
    section ✅ merge-ready 4m 20s
    ✅ :active, m, 2000-01-01 00:04:20, 2000-01-01 00:05:38
Loading

⏳: job creation-to-start delay, including scheduling/admission. Orange: the critical path through the workflow's needs graph. Bars are observed API timestamps.

Critical path

Observed dependency chain (4m 20s): select-tests → saleor → Validation gate

Execution-only CPM (4m 14s): select-tests → saleor → Validation gate

Job Execution (s) Delay after prerequisites (s) Execution slack (s)
select-tests 13 2 0
lint 12 2 239
ci-contracts 26 2 212
typecheck 16 2 222
core (3.11) 121 2 117
core (3.12) 201 2 37
core (3.13) 138 3 100
core (3.14) 142 3 96
minimum-models 146 2 92
exploration 157 3 81
django 68 2 170
django-psycopg2 62 2 176
redis 151 2 87
mongodb 35 2 203
aiokafka 45 2 193
prefect 108 2 130
demos 236 2 2
saleor 238 2 0
build 16 2 222
actions 38 2 200
due 174 2 64
Validation gate 3 2 0

Delay includes scheduling, runner admission and other waits; it is not isolated queue time. Execution CPM holds measured durations constant and excludes those delays. The declared DAG must cover every prerequisite; this analysis does not assert validation success.

Runner minutes and cost by runner and job

Estimated runner usage for this run: 48 rounded min from 35.8 raw min; 12.2 min (25%) is per-job rounding.

Estimated runner cost at recorded rates: $0.

runner SKU rounded min USD/min estimated cost
github_public_actions_linux 48 $0 $0

Hosted runner rates: GitHub pricing, verified 2026-10-24. List-price estimates are before included minutes, discounts and taxes; they are not an invoice. Known public repositories use the free standard-runner rate; larger runners remain billable. Explicit configured rates take precedence.

job runner labels runner / group attempt wall rounded min save 1 min headroom
saleor ubuntu-latest GitHub Actions 1000528395 / GitHub Actions 1 238s 4 58s 2s
demos ubuntu-latest GitHub Actions 1000528396 / GitHub Actions 1 236s 4 56s 4s
core (3.12) ubuntu-latest GitHub Actions 1000528394 / GitHub Actions 1 201s 4 21s 39s
due ubuntu-latest GitHub Actions 1000528386 / GitHub Actions 1 174s 3 54s 6s
exploration ubuntu-latest GitHub Actions 1000528381 / GitHub Actions 1 157s 3 37s 23s
redis ubuntu-latest GitHub Actions 1000528389 / GitHub Actions 1 151s 3 31s 29s
minimum-models ubuntu-latest GitHub Actions 1000528385 / GitHub Actions 1 146s 3 26s 34s
core (3.14) ubuntu-latest GitHub Actions 1000528387 / GitHub Actions 1 142s 3 22s 38s
core (3.13) ubuntu-latest GitHub Actions 1000528383 / GitHub Actions 1 138s 3 18s 42s
core (3.11) ubuntu-latest GitHub Actions 1000528393 / GitHub Actions 1 121s 3 1s 59s
prefect ubuntu-latest GitHub Actions 1000528391 / GitHub Actions 1 108s 2 48s 12s
django ubuntu-latest GitHub Actions 1000528382 / GitHub Actions 1 68s 2 8s 52s
django-psycopg2 ubuntu-latest GitHub Actions 1000528380 / GitHub Actions 1 62s 2 2s 58s
aiokafka ubuntu-latest GitHub Actions 1000528378 / GitHub Actions 1 45s 1 remove/consolidate 15s
actions ubuntu-latest GitHub Actions 1000528390 / GitHub Actions 1 38s 1 remove/consolidate 22s
mongodb ubuntu-latest GitHub Actions 1000528392 / GitHub Actions 1 35s 1 remove/consolidate 25s
ci-contracts ubuntu-latest GitHub Actions 1000528384 / GitHub Actions 1 26s 1 remove/consolidate 34s
typecheck ubuntu-latest GitHub Actions 1000528388 / GitHub Actions 1 16s 1 remove/consolidate 44s
build ubuntu-latest GitHub Actions 1000528379 / GitHub Actions 1 16s 1 remove/consolidate 44s
select-tests ubuntu-latest GitHub Actions 1000528376 / GitHub Actions 1 13s 1 remove/consolidate 47s
lint ubuntu-latest GitHub Actions 1000528374 / GitHub Actions 1 12s 1 remove/consolidate 48s
Validation gate ubuntu-latest GitHub Actions 1000528458 / GitHub Actions 1 3s 1 remove/consolidate 57s

The organization billing ledger is authoritative; this estimate applies a per-job minute ceiling to GitHub timestamps.

Slowest step in each job
job result job time slowest timed step step time
saleor success 238s (complete) Saleor checkout 181s (complete)
demos success 236s (complete) Wagtail on django-tasks-db 74s (complete)
core (3.12) success 201s (complete) Prove package-only core controls need no application checkout 90s (complete)
due success 174s (complete) Run complete work-table and application conformance 93s (complete)
exploration success 157s (complete) Run uv run --no-sync pytest package/tests/core integration_tests/core 142s (complete)
redis success 151s (complete) Run uv run --no-sync pytest package/tests/rq package/tests/celery_worker integration_tests/rq integration_tests/celery_worker -p no:django -rxX -n 4 --durations=10 --due-work-explore=smoke --due-work-check-fixtures --due-work-require-assessed --due-work-profile-report=profiles-redis.json --junitxml=adopters-redis.xml 128s (complete)
minimum-models success 146s (complete) Run uv run --no-sync pytest package/tests/core integration_tests/core 126s (complete)
core (3.14) success 142s (complete) Run uv run --python 3.14 --no-sync pytest package/tests/core integration_tests/core --junitxml=test-timings.xml -o junit_family=xunit1 128s (complete)
core (3.13) success 138s (complete) Run uv run --python 3.13 --no-sync pytest package/tests/core integration_tests/core --junitxml=test-timings.xml -o junit_family=xunit1 117s (complete)
core (3.11) success 121s (complete) Run uv run --python 3.11 --no-sync pytest package/tests/core integration_tests/core --junitxml=test-timings.xml -o junit_family=xunit1 105s (complete)
prefect success 108s (complete) Run uv run --no-sync pytest package/tests/prefect integration_tests/prefect -q --due-work-explore=smoke --due-work-check-fixtures --due-work-require-assessed --due-work-profile-report=profiles-prefect.json --junitxml=adopters-prefect.xml --durations=20 96s (complete)
django success 68s (complete) Run uv run --no-sync pytest package/tests/django examples/adopter/tests_django --ds=tests.django.settings -n 4 38s (complete)
django-psycopg2 success 62s (complete) Run uv run --no-sync pytest package/tests/django examples/adopter/tests_django --ds=tests.django.settings -n 4 29s (complete)
aiokafka success 45s (complete) Initialize containers 22s (complete)
actions success 38s (complete) Generated histories also support optional search 6s (complete)
mongodb success 35s (complete) Run uv run --no-sync pytest package/tests/mongodb -q 13s (complete)
ci-contracts success 26s (complete) Prove consumer routing and documentation checks 16s (complete)
typecheck success 16s (complete) Run uv sync --all-packages --locked --all-extras --group demos 3s (complete)
build success 16s (complete) Verify old and new adopters against the installed wheel 3s (complete)
select-tests success 13s (complete) Run gigaverse-app/pytest-selective/actions/plan-lanes@v1 5s (complete)
lint success 12s (complete) Run gigaverse-app/ci-esta/actions/setup-cached-uv@v1 4s (complete)
Validation gate success 3s (complete) Set up job 1s (complete)

Latest attempt only. All observed steps are retained in the JSON report. Unavailable timestamps are not zero; accrued time is measured at the snapshot. Step times explain job time and are not added to runner usage.


Savings across selection, reuse, and caching are not additive.

CI-esta · Let proven work rest. Measure the rest.

Reading this report · run 40179895071 · schema ci-observability/v1

@aviadr1
aviadr1 changed the base branch from claude/celery-table-after-fixes to main October 25, 2026 09:03
aviadr1 and others added 4 commits October 25, 2026 11:03
…histories

# Conflicts:
#	package/tests/celery_worker/test_celery_worker_contract.py
…act's design checks

ObligationContract already refuses duplicate handoff history names and
handoff_gaps that name no history; worker_contract no longer repeats them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…tory=

worker_contract takes the adopter's histories as ObligationContract does,
process_handoffs= with handoff_gaps= by history name, replacing the
unreleased histories=. history= and gap= still bind and now warn.

The reference test builds both histories through one helper, and gates
the rows the #10724 fixes remove through one helper with its retirement
condition. The docs describe the binding beside the integrations table.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@aviadr1 aviadr1 changed the title feat(celery): bind several worker histories in one contract feat(celery): bind worker histories with process_handoffs= and deprecate history= Oct 25, 2026
@aviadr1
aviadr1 merged commit d2b8ac2 into main Oct 25, 2026
26 checks passed
github-actions Bot pushed a commit that referenced this pull request Oct 25, 2026
…ate history= (#161)

## TLDR

Today `celery_worker.worker_contract()` binds exactly one history, so an
adopter checking a task and a chord has to `model_copy` the second
history onto `process_handoffs` and `handoff_gaps` by hand.
celery/celery#10726 does exactly that. This PR makes `worker_contract`
take its histories the way `ObligationContract` itself does:
`process_handoffs=` with `handoff_gaps=` keyed by history name, the same
field names with the same meaning, so nothing is translated between two
vocabularies. The one-history form, `history=` with `gap=`, is
deprecated: it still binds, and emits a `DeprecationWarning` naming the
replacement. A mix of the two forms is refused in words.

The reference self-test now binds a chord of two `send_message` tasks
next to the single task. On the locked Celery 5.6.3, that chord
reproduces celery/celery#10724's chord case on a real worker: a header
lost after its SUCCESS was stored, or whose `on_success` raised, fails
the whole chord.

Builds on #160 (merged), which gates the fixed rows on the Celery
release.

```text
BEFORE                                            AFTER
worker_contract(history=TASK, gap=...)            worker_contract(process_handoffs=(TASK,), handoff_gaps={TASK.name: ...})
                                                  (history=/gap= still work and warn: DeprecationWarning)
a second history: model_copy(update={             worker_contract(process_handoffs=(TASK, CHORD),
  "process_handoffs": (..., CHORD),                   handoff_gaps={TASK.name: ..., CHORD.name: ...})
  "handoff_gaps": {..., CHORD.name: ...}})
```

## What the chord history pins

The faults hit the first execution of a header task. The refused publish
is the last header publishing the body.

| History | Celery ≤ 5.7.0rc1 (lock 5.6.3) | Later releases (verified on
Celery `main`) |
| --- | --- | --- |
| normal operation | SUCCESS, 2 sent, `sent 2 messages` | same |
| died at `task_prerun` | FAILURE, 1 sent, `failed: ChordError` | same |
| died at `mark_as_done` | FAILURE, 2 sent, `failed: ChordError`
(celery/celery#10949) | same |
| died at `task_postrun` | FAILURE, 2 sent, `failed: ChordError`
(celery/celery#10724) | normal operation |
| `on_success` raised | FAILURE, 2 sent, `failed: ChordError`
(celery/celery#10724) | normal operation |
| broker refused the body | FAILURE, 2 sent, `failed: ChordError`
([#10725
comment](celery/celery#10725 (comment)))
| same |

## Refused bindings

`worker_contract` asserts, naming the contract:
- not both the deprecated `history=` and `process_handoffs=` or
`handoff_gaps=`;
- no `gap=` without `history=`;
- at least one history.

Two histories sharing a name, and a `handoff_gaps` key naming no bound
history, are refused by `ObligationContract`'s own design checks
(`ObligationContractDesignError`), so `worker_contract` doesn't repeat
them; a test pins that they apply to several histories.

## Reference test and docs

- Both histories are built by one `reference_history(name, send,
findings)` helper instead of two near-identical `worker_history` calls.
- The rows #10752/#10759 remove are added through one
`before_the_10724_fixes(rows)` helper beside the gate, which now states
its retirement condition: delete it and those rows once `uv.lock`'s
Celery is newer than 5.7.0rc1.
- `package/docs/integrations.md` describes the binding beside the
integrations table, with an example and the deprecation.

## Tested

WSL, Python 3.12, Redis 7.4.11, `package/tests/celery_worker -p
no:django -rxX`:

| Celery | Result |
| --- | --- |
| 5.6.3 (the lock) | 15 passed, 3 xfailed |
| `main` at e3cc7cc53, release gate forced on | 15 passed, 3 xfailed |

- `test_celery_worker_contract_arguments.py`: 8 passed (the binding, the
deprecated form binding with its warning, 4 mixed-form refusals, 2 core
design refusals).
- After these changes, `package/tests/celery_worker` on Celery 5.6.3
with real workers: 23 passed, 3 xfailed. It needs no worker, so it also
runs on Windows.
- `ruff check package` and `ruff format --check package`: clean.
- `pyrefly check`: no errors in `celery_worker` files.

The 5.6.3 run shows 7 `DeprecationWarning`s from Celery's own Redis
backend (`setex`) on the chord path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
aviadr1 added a commit that referenced this pull request Oct 25, 2026
The fault commits were written over an older copy of #161 and reverted its
process_handoffs= binding, the history= deprecation and the move of name
checks into the contract. This restores them with a three-way merge and
binds the SIGKILL and full-broker histories through reference_history.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
aviadr1 added a commit that referenced this pull request Oct 25, 2026
…broker (#165)

## TLDR

Today the Celery worker integration fakes the two failures it is about.
A "death" is the pool child calling `os._exit(1)` from inside a hook,
and a "refused link" is a patched `Signature.apply_async` raising
kombu's `OperationalError`. Both are realistic: a probe against
celery/celery#10726 got identical outcomes from a real outside `kill -9`
and from a real Redis broker full under `noeviction`. But that probe was
a one-off script, and Celery reports a signal death through a different
path from an exit status.

This PR makes the real failures part of the integration, so every
contract can run them:
- **`SIGKILL_DEATH_POINTS`:** the kernel kills the pool child at the
same three stages.
- **`BROKER_FULL`:** the broker's own Redis server refuses writes once
the task's body has run.

Both are opt-in, so no adopter's pinned findings table changes. The
reference contract runs both, on Celery 5.6.3 and on Celery `main`.

**Stacked on #161** (and #160). Review the last three commits.

## The fault list

```text
BEFORE                                         AFTER
deaths (os._exit(1) in the pool child)         deaths (os._exit(1)), default, unchanged
  task_prerun, mark_as_done, task_postrun        task_prerun, mark_as_done, task_postrun
                                               SIGKILL_DEATH_POINTS (kernel SIGKILL), opt-in
                                                 task_prerun (SIGKILL), mark_as_done (SIGKILL), task_postrun (SIGKILL)
failures the worker survives, default          failures the worker survives, default, unchanged
  the task's on_success hook raised              the task's on_success hook raised
  the broker refused the task's link             the broker refused the task's link (patched apply_async)
    (patched apply_async)                      BROKER_FULL, opt-in, needs broker_url=
                                                 the Redis broker was full (CONFIG SET maxmemory 1, noeviction)
```

## Why opt-in

Every adopter's `Findings` table pins every history it runs. A new
default death point would show up in their tables as an unpinned history
and turn their suites red on upgrade. With opt-in, an adopter adds them
when it chooses:

```python
worker_history(..., death_points=DEATH_POINTS + SIGKILL_DEATH_POINTS)
worker_history(..., death_points=(), failure_points=(BROKER_FULL,), broker_url=OWN_BROKER)
```

`BROKER_FULL` also needs a broker on a Redis server shared with nothing
else: `maxmemory` is server-wide, so the result backend and anything the
test reads must live elsewhere.

## How the broker-full fault is contained

- **In the worker's pool child:** after the task's body returns, on its
first execution only, the child sets the broker's server to `maxmemory
1` with `noeviction`. Redis then refuses the writes that follow, and
kombu re-raises Redis's `OutOfMemoryError` as `OperationalError`.
- **In the test process:** `broker_restored(broker_url)` reads
`maxmemory` and `maxmemory-policy` before the run and restores them on
exit, before the worker stops and before recovery. That happens whether
the run settled, failed or timed out.
- **Refused in words:** a missing or non-Redis `broker_url` at
declaration, a non-Redis broker in the worker, and a server that refuses
`CONFIG`.

## What the reference contract pins

WSL, Python 3.12, Redis 7.4. The broker was a separate Redis container.
`package/tests/celery_worker -p no:django -rxX`.

| History | Celery 5.6.3 (the lock) | Celery `main` e3cc7cc53 (gate
forced on) |
| --- | --- | --- |
| died at `task_prerun (SIGKILL)` | FAILURE, 0 sent, `failed:
WorkerLostError` | same |
| died at `mark_as_done (SIGKILL)` | FAILURE, 1 sent, `failed:
WorkerLostError` + `sent` (celery/celery#10949) | same |
| died at `task_postrun (SIGKILL)` | SUCCESS, 1 sent, `failed:
WorkerLostError` + `sent` (celery/celery#10724) | normal operation
(fixed by #10752) |
| the Redis broker was full | FAILURE, 1 sent, `failed:
OperationalError`, link never sent (celery/celery#10725) | same |

Each SIGKILL row matches its `os._exit` row on the same Celery version.
The full broker matches the injected refusal row. So the injected faults
were faithful, and the suite will notice if Celery's signal path or its
refusal handling ever diverges from them.

| Run | Result |
| --- | --- |
| Celery 5.6.3 | 23 passed, 4 xfailed; broker server's `maxmemory` back
to `0` afterwards |
| Celery `main`, gate forced on | 23 passed, 4 xfailed |
| `test_celery_worker_broker_full.py` | 3 passed: refusals in words, and
the restore when the run raises |

- `ruff check`, `ruff format --check`: clean.
- `pyrefly check`: no errors in Celery files.

## CI

The `redis` job gains a second service, `redis-own-broker` on port 6380,
and `CELERY_OWN_BROKER_URL`. Only the broker-full history uses it, so
filling it cannot refuse writes for the other xdist workers' suites.
`test_the_broker_is_restored_however_the_run_ends` changes the shared
server's `maxmemory` only to 512 MiB, never to a value that refuses
writes.


**Stacked on #161 and built on its API.** The first commits were written
over an older copy of #161 and reverted its `process_handoffs=` binding
and `history=` deprecation; fb33765 restores them by three-way merge.
Since then the new histories are bound through #161's
`reference_history` and `process_handoffs=`. Relative to #161 this PR
now only adds: 26 passed, 4 xfailed locally on Celery 5.6.3, with no
`history=` deprecation warning raised.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
aviadr1 added a commit that referenced this pull request Oct 25, 2026
This is the manual release decision. Leave this PR open to accumulate
alpha changes without publishing.

**CI approval only runs tests; it does not publish. Merging this PR
authorizes publication:** after normal CI passes on the merged commit,
the Release workflow tags, builds and publishes that exact version to
PyPI.

If GitHub asks to approve workflows on each update, the release bot is
using GITHUB_TOKEN. The configured GV_CI_CROSS_REPO_TOKEN avoids that
prompt; see RELEASING.md.
---


##
[0.20.0](v0.19.2...v0.20.0)
(2026-10-25)


### Features

* **celery:** bind worker histories with process_handoffs= and deprecate
history=
([#161](#161))
([d2b8ac2](d2b8ac2))
* **report:** publish fork PR evidence with an isolated renderer
([#162](#162))
([6936ca6](6936ca6))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
aviadr1 added a commit that referenced this pull request Oct 25, 2026
…broker (land #165 on main) (#172)

## TLDR

#165 (SIGKILL deaths and a real full Redis broker for the Celery worker
integration) shows as merged, but its base was #161's branch, and it
merged there 17 minutes after #161 itself had merged to `main`. Its
change never reached `main`, so 0.20.0 has neither
`SIGKILL_DEATH_POINTS` nor `BROKER_FULL`. This PR carries #165's diff
(its merge commit f8a7b50 against its first parent) onto `main`
unchanged.

```text
BEFORE   #161 ──merged──> main ──> 0.20.0          #165 ──merged──> claude/celery-worker-histories (dead branch)
AFTER    #161 + this PR ──> main ──> next release   (SIGKILL_DEATH_POINTS, BROKER_FULL, broker_restored)
```

What it adds is described in #165: all opt-in, so no adopter's findings
table moves.

## Tested

On this branch, in WSL with Python 3.12, Redis 7.4.11 and a second Redis
for the full-broker history, `package/tests/celery_worker`: 26 passed, 4
xfailed. `ruff check` and `ruff format --check`: clean.

Found while building a chord contract on gigaverse-app/celery: `from
pytest_obligation.integrations.celery_worker import
SIGKILL_DEATH_POINTS` fails on 0.20.0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
github-actions Bot pushed a commit that referenced this pull request Oct 25, 2026
…broker (land #165 on main) (#172)

## TLDR

#165 (SIGKILL deaths and a real full Redis broker for the Celery worker
integration) shows as merged, but its base was #161's branch, and it
merged there 17 minutes after #161 itself had merged to `main`. Its
change never reached `main`, so 0.20.0 has neither
`SIGKILL_DEATH_POINTS` nor `BROKER_FULL`. This PR carries #165's diff
(its merge commit f8a7b50 against its first parent) onto `main`
unchanged.

```text
BEFORE   #161 ──merged──> main ──> 0.20.0          #165 ──merged──> claude/celery-worker-histories (dead branch)
AFTER    #161 + this PR ──> main ──> next release   (SIGKILL_DEATH_POINTS, BROKER_FULL, broker_restored)
```

What it adds is described in #165: all opt-in, so no adopter's findings
table moves.

## Tested

On this branch, in WSL with Python 3.12, Redis 7.4.11 and a second Redis
for the full-broker history, `package/tests/celery_worker`: 26 passed, 4
xfailed. `ruff check` and `ruff format --check`: clean.

Found while building a chord contract on gigaverse-app/celery: `from
pytest_obligation.integrations.celery_worker import
SIGKILL_DEATH_POINTS` fails on 0.20.0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant