Read the documentation · Related tools
PR evidence report: what passed, what is missing, what to fix
Your app may commit an order, charge a customer, schedule a task, send a receipt, publish to Kafka, or call Shopify. A worker can die one line later; a broker reply can be lost; a retry can run the external effect twice. Ordinary tests usually exercise only the path where none of that happens.
pytest-obligation generates tests for work your application cannot afford to
lose or repeat. Bind an ObligationContract to your real workflow and recovery
path, yourself or with a coding agent. The harness generates standardized tests
for crashes, retries, lost messages, and competing operations, exposing failure
cases you might never think to write by hand. Its fault injection interrupts
commits, workers, messages, and external calls, then checks whether recovery
reaches the right outcome without lost work or duplicate effects.
The agent can use the same fault-injection infrastructure to add focused tests
for your application's particular risks. The core accepts plain Python
callables: it assumes no framework, database, queue, or business domain.
It integrates as a pytest plugin in your existing test suite and CI. Read the pytest plugin guide for automatic discovery, test selection, reports, and configuration.
Coming from distributed systems and familiar with Jepsen? pytest-obligation brings a similar approach to your application's workflows. Read Jepsen-style fault testing for application work.
- Lost work: a Django transaction commits an order, but a Celery task or Kafka message is never published; a dead worker's task is never picked up.
- Duplicate effects: a retry sends two receipts, charges twice, or repeats an external API call after the first result became uncertain.
- Wrong final state: a stale worker overwrites a newer result, a checkout takes a payment without creating an order, or cleanup deletes work still owed.
These are not just hypothetical failures. Runs against Saleor, Wagtail, procrastinate, DBOS, RQ, and Celery show the real code, failure timing, observed result, and—in several cases—a fix that makes the same proof pass. For example, a Saleor Payments API checkout can capture a payment and lose the order when the worker dies between those steps; a DBOS notification step can send twice when it is replayed.
One crash history compares the normal outcome with the outcome after a specific interruption:
pytest_obligation.HistoriesDiverged: orders: handoff 'place order': normal operation reaches
('SENT', 1), but these histories reach something else: {'worker died after external call 1':
('SENT', 2)}. Work was lost or repeated. ...
The second send is visible instead of being hidden behind a green happy-path test. Known, independently reproduced gaps remain strict xfails; a change that fixes one turns it red so the finding must be re-evaluated. See how crash histories work and what a green result means.
Declare the workflow once in an ObligationContract and expose it through
@due_work_contract_suite(CONTRACT). At pytest collection time, the harness
generates cases for the applicable A–J guarantees: recovery, ownership, crash
ambiguity, retention, convergence, derived obligations, gated execution,
harmless replay, indivisible admission, and retry limits. As the harness gains
checks for a profile you already bound, upgrading can generate more cases from
the same contract. New profiles or bindings still need your assessment. The
separate handoff scan spots newly introduced queue/task handoffs that need a
contract; it does not silently claim they are proven. See the guarantees and
limits.
pytest --due-work-summary lists generated cases, passes, and declared gaps:
test_wagtail_tasks.py::TestDjangoTasksDb: 7 passed, 5 xfailed
XFAIL A-known_gap (the worker selects only READY tasks, so a task left RUNNING by a worker that died is…)
PASSED D-assert_retention_preserves_non_terminal_work
...
For CI, --due-work-profile-report=profiles.json records what was declared,
collected, selected, and actually executed. An xfail or unassessed profile is
not a verified guarantee. Read the reporting guide.
Install pytest-obligation from PyPI in your application's test environment.
The former due-work-harness distribution should be removed when migrating;
the migration guide explains compatibility.
The --due-work-* options, due-work-harness command, and
[tool.due-work-harness] configuration remain supported.
See the compatibility and migration guide.
Recovery loops also have poison isolation, cooperative-stop and database-legal row checks.
The complete backend harness port includes Due work-table conformance and reusable source-policy checks. The cutover follow-through audit records the remaining improvements and their public test owners.
Install the Python test library in the application you want to test (Python 3.11+). Choose your package manager; add an optional extra only for an integration you use:
pip install pytest-obligation
uv add --dev pytest-obligation
poetry add --group dev pytest-obligationOptional integrations:
For provider simulation, see the dependency-free Shopify billing simulator: accepted requests, permanently consumed keys, rejection and delayed meter evidence.
For example, use "pytest-obligation[django]" with any of the commands above
for the Django/PostgreSQL integration. Other extras include [celery], [rq],
[mongodb], [prefect], and [aiokafka]; see integration scope.
The current source has A–J profiles. Check the adoption guide matching your
installed version; older PyPI releases use an earlier profile catalog.
Then add the coding-agent plugin (it provides instructions, not the Python library). For Claude Code:
claude plugin marketplace add gigaverse-app/pytest-obligation
claude plugin install pytest-obligation@pytest-obligationFor Codex, add the repository marketplace, then open the Plugins Directory, select pytest-obligation, and install the plugin:
codex plugin marketplace add gigaverse-app/pytest-obligation --sparse .agents/pluginsIt is not yet in the public plugin directory. The plugin works without a remote MCP server; tests run in your coding environment with your app and its required services.
Open your application in Claude Code or Codex and ask:
Use the
prove-due-workskill to create anObligationContractfor our order-processing workflow. Bind the real database-to-queue handoff and recovery worker, generate and run the standard tests, then add focused crash/retry tests. Report what passed, what failed, and which guarantees remain unassessed.
| You want to… | Read |
|---|---|
| Learn by doing, step by step | Tutorials: workflows across many systems, for framework authors, vocabulary |
| Find the right workflow or public interface | Harness guide and module map |
| Bind your application and run generated proofs | Adoption walkthrough, runnable SQLite adopter |
| Bind a Due work-table application | Due's conformance guide |
| Inspect transaction and schedule policy | Handoff coverage, source-policy APIs and examples |
| Understand recovery and outcome evidence | Recovery edge cases, what a green result means |
| Enforce known gaps and selection completeness | Debt baseline, selection completeness |
| Hold a path's database round trips to a baseline | Cost contracts |
| Probe another project | Upstream playbook, framework integrations |
| Replace the backend harness | Completed port and ownership map |
| Edit the library inside an adopter checkout | Package ownership and subtree development |
| Contribute a proof or integration | Architecture, contributor workflow, CI selection and costs |
Alpha; the public API may change before 1.0. Architecture · Contributing · Apache-2.0 license
Pull requests run only the test lanes a change can affect: a package README edit skips every live framework lane. Every run gets a CI-esta timing report.
See the CI toolkit integration for lane selection, validated environment caches, required gates and real dependency critical-path reports.
Report merging, cost-baseline retirement and profile B's reference owner are each checked against a Quint model on every relevant pull request: generated histories are replayed through the real code, and deliberately planted bugs must be caught. The contracts, what they model and the bugs they found.
✅ 3 contracts · 82/82 cases · 9/9 mutants caught · 70 histories
| Cases passed | Mutants caught | ||
|---|---|---|---|
| ✅ | cost-baseline pytest_obligation/_cost_plugin/cost_baseline/test_cost_baseline_contract.py::test_cost_baseline26 histories (10 witnesses, 16 samples, seed 1) · every action covered · Quint 0.33.1 |
30/30 | 3/3 |
| ✅ | fenced-lease pytest_obligation/references/in_memory/fenced_lease/test_fenced_lease_contract.py::test_fenced_lease23 histories (5 witnesses, 18 samples, seed 20261108) · every action covered · Quint 0.33.1 |
27/27 | 3/3 |
| ✅ | report-merge pytest_obligation/profiles/reporting/report_merge/test_report_merge_contract.py::test_report_merge21 histories (5 witnesses, 16 samples, seed 20261107) · every action covered · Quint 0.33.1 |
25/25 | 3/3 |
🐞 3 real bugs found by these contracts
| Contract | Fixed in | Regression test | |
|---|---|---|---|
cost-baseline |
Running with OBLIGATION_COST_BASELINE=update after an incomplete run (failed collection, deselection, early exit, failure, collect-only or xdist) deleted every baseline entry that run did not measure found by the historical audit model (audit.qnt, pinned to 389c5bb): baselineDeletionWitnessTest reaches the deletion after a collection error · mutant shipped-389c5bb reintroduces it: 22 of 26 histories diverge |
#208 | package/tests/core/test_costs.py::test_incomplete_update_does_not_retire_unmeasured_contracts |
fenced-lease |
Profile B's reference owner kept a reaped owner's token valid until the next claim, so that owner could still renew a READY row or write it to RETRYABLE, where no claim or reaper reaches it; the profile's stale-token proofs never tried the token in that window found by a state-machine audit of the references, confirmed by replaying claim, expire, reclaim and a late write against the real InMemoryOwner · mutant shipped-9aa6ff2 reintroduces it: 5 of 23 histories diverge |
#217 | package/tests/core/profiles/test_bounded_ownership.py::test_a_reaped_owner_cannot_touch_the_row_before_it_is_claimed_again |
report-merge |
A rejected report shard leaked its earlier cases and new suites into the combined report, which then showed that refused evidence as available found by the historical audit model (audit.qnt, pinned to 389c5bb): its partialPromotionWitnessTest reaches the leak · mutant shipped-389c5bb reintroduces it: 16 of 21 histories diverge |
#210 | package/tests/core/test_profile_coverage.py::test_rejected_shard_cannot_promote_or_mutate_any_prior_evidence |
🧪 9 mutants (planted fake bugs), all caught
| Contract | Mutant | Caught by | At | |
|---|---|---|---|---|
cost-baseline |
shipped-389c5bb |
The exact CostCompleteness.pytest_sessionfinish shipped at 389c5bb, before the fix: update mode prunes every unmeasured entry at session end, even after a failed collection, filter or crash reintroduces a real bug found earlier |
witness-failedCollectionPreservesTest |
state 2, after finishUpdate(collected=0, status=2) |
cost-baseline |
ignore-deselection |
Deselected tests are not recorded as incomplete, so a filtered run can retire entries its deselected tests would have measured | witness-skippedRunPreservesTest |
state 2, after finishUpdate(collected=1, status=0) |
cost-baseline |
disable-retirement |
Pruning never happens, so a complete run cannot retire a contract no test measures any more | witness-retirementControlTest |
state 2, after finishUpdate(collected=1, status=0) |
fenced-lease |
shipped-9aa6ff2 |
The exact InMemoryOwner re-arm methods shipped at 9aa6ff2, before the fix: the row returns to READY but keeps the previous owner's token, which can still write or renew it reintroduces a real bug found earlier |
witness-reapedOwnerCannotWriteTest |
state 4, after write(token='last') |
fenced-lease |
reaper-ignores-lease |
The reaper re-arms any claimed row, taking it from an owner whose lease is still live | witness-reapedOwnerCannotWriteTest |
state 3, after reclaim(token='') |
fenced-lease |
unfenced-renewal |
renew_lease ignores the token, so a dead owner can keep a lease alive | witness-reapedOwnerCannotRenewTest |
state 4, after renew(token='last') |
report-merge |
shipped-389c5bb |
The exact merge_reports shipped at 389c5bb, before the fix: it writes a shard's earlier cases and new suites into the report before a later mismatch rejects the shard reintroduces a real bug found earlier |
witness-repeatedLateRejectionTest |
state 1, after reject(location='later-suite', mismatch='family', payload='passed') |
report-merge |
incremental-admission |
Today's merge_reports with its commit loop moved back inside validation, so the same bug returns even after other edits to the function | witness-repeatedLateRejectionTest |
state 1, after reject(location='later-suite', mismatch='family', payload='passed') |
report-merge |
replace-case-identity |
After a compatible merge, a retained case is swapped for an equal copy, so code holding the old object stops seeing new evidence | witness-compatibleRepeatedEvidenceTest |
state 1, after merge(location='later-suite', mismatch='', payload='passed') |
What cost-baseline says
Actions, and the code each one drives:
init: Create actual pytest Config/Session, CostCompleteness, and two-entry baseline file.collectionError: CostCompleteness.pytest_collectreport with actual failed pytest.CollectReport.skipOrDeselect: Passed pytest.TestReport phases followed by pytest_deselected with a pytest.Item.earlySuccessfulExit: One actual completed TestReport sequence; finish input says two collected tests.testFailure: Actual failed call report and passed teardown; final TESTS_FAILED status.collectOnly: Instantiate CostCompleteness with Config.option.collectonly.distributedRun: Instantiate with Config.option.numprocesses=2; actual measured TestReport sequence.setupOnly: Instantiate with Config.option.setuponly; actual setup and teardown TestReports with no call report.lastFailed: Instantiate with Config.option.lf; actual measured TestReport sequence, as for the one file --lf kept.completeRun: Actual passed TestReport sequence with obligation_cost_contract property.emptyCollection: No report; final NO_TESTS_COLLECTED status.finishUpdate: Call real CostCompleteness.pytest_sessionfinish with actual pytest.Session inputs.inspect: Observe baseline via load_cost_baseline and real plugin visited/finished/passed_calls fields; no repeated hook.
Invariant incompletePreservesBaseline, checked while sampling histories.
Witnesses, scenarios replayed exactly as written: failedCollectionPreservesTest, skippedRunPreservesTest, earlyExitPreservesTest, failurePreservesTest, collectionOnlyPreservesTest, distributedRunPreservesTest, setupOnlyPreservesTest, lastFailedPreservesTest, retirementControlTest, emptyRetirementControlTest.
Bounds: seed 1 · max_samples 64 · max_steps 6 · n_traces 16.
Witness failedCollectionPreservesTest, step by step: the observation the model starts from, then what it expects each action to change.
| State | Action | Model expects |
|---|---|---|
0 |
init(collected=0, status=0) |
{ baseline: Set("required", "retired"), finished: 0, passedCalls: 0, visited: Set() } |
1 |
collectionError(collected=0, status=2) |
unchanged |
2 |
finishUpdate(collected=0, status=2) |
unchanged |
3 |
inspect(collected=0, status=2) |
unchanged |
What fenced-lease says
Actions, and the code each one drives:
init: A fresh InMemoryOwner and one claimable rowclaim: InMemoryOwner.claim(); the returned token is remembered as the last issuedexpire: InMemoryOwner.expire_lease(row): the owner stopped renewingreclaim: InMemoryOwner.reclaim_stalled(row): the reaperwrite: InMemoryOwner.fenced_write(row, token) with the last or previous issued tokenretry: InMemoryOwner.fenced_retry(row, token) with the last or previous issued tokenrelease: InMemoryOwner.release_unchanged(row, token) with the last or previous issued tokenrenew: InMemoryOwner.renew_lease(row, token) with the last or previous issued tokeninspect: Read InMemoryOwner.observe(row) without changing it
Invariant contract, checked while sampling histories.
Witnesses, scenarios replayed exactly as written: reapedOwnerCannotWriteTest, reapedOwnerCannotRenewTest, supersededOwnerCannotWriteTest, currentOwnerWritesTest, liveLeaseBlocksReclaimTest.
Bounds: seed 20261108 · max_samples 200 · max_steps 8 · n_traces 18.
Witness reapedOwnerCannotWriteTest, step by step: the observation the model starts from, then what it expects each action to change.
| State | Action | Model expects |
|---|---|---|
0 |
init(token='') |
{ accepted: false, lease: "none", state: "READY" } |
1 |
claim(token='') |
accepted: false → truelease: "none" → "live"state: "READY" → "CLAIMED" |
2 |
expire(token='') |
accepted: true → falselease: "live" → "expired" |
3 |
reclaim(token='') |
accepted: false → truestate: "CLAIMED" → "READY" |
4 |
write(token='last') |
accepted: true → false |
5 |
inspect(token='') |
unchanged |
What report-merge says
Actions, and the code each one drives:
init: Construct real SuiteCoverage / ProfileCoverage / CaseEvidence fixtures; no production merge guard patchedmerge: profiles.reporting.merge_reports(target, compatible shard), direct callreject: profiles.reporting.merge_reports(target, shard with indicated late mismatch), observe actual AssertionError and stateinspect: SuiteCoverage.verified, CaseEvidence.outcomes and retained object references read without mutation
Invariant contract, checked while sampling histories.
Witnesses, scenarios replayed exactly as written: repeatedLateRejectionTest, compatibleRepeatedEvidenceTest, lateRejectionRetainsPriorSuccessTest, failedRepeatCannotBecomeVerifiedTest, interruptedRepeatRevokesSuccessTest.
Bounds: seed 20261107 · max_samples 150 · max_steps 6 · n_traces 16.
Witness repeatedLateRejectionTest, step by step: the observation the model starts from, then what it expects each action to change.
| State | Action | Model expects |
|---|---|---|
0 |
init(location='', mismatch='', payload='') |
{ extraExists: false, extraFirst: { bad: false, call: 0, collected: false, durationTicks: 0, failureCount: 0, selected: false, setup: 0, teardown: 0 }, extraSecond: { bad: false, call: 0, collected: false, durationTicks: 0, failureCount: 0, selected: false, setup: 0, teardown: 0 }, extraVerified: false, identityRetained: true, laterFirst: { bad: false, call: 0, collected: false, durationTicks: 0, failureCount: 0, selected: false, setup: 0, teardown: 0 }, laterSecond: { bad: false, call: 1, collected: true, durationTicks: 3, failureCount: 0, selected: true, setup: 1, teardown: 1 }, laterVerifi… |
1 |
reject(location='later-suite', mismatch='family', payload='passed') |
rejected: false → true |
2 |
reject(location='later-suite', mismatch='debt', payload='passed') |
unchanged |
3 |
reject(location='same-suite', mismatch='declaration', payload='passed') |
unchanged |
4 |
inspect(location='', mismatch='', payload='') |
unchanged |
Generated by pytest-quint-connect from contract receipts; what a passing run proves.
The local hooks and required PR check prevent new copies; the weekly audit records progress against the accepted backlog. Contributor setup and cleanup workflow.
Reviewed duplication debt · Counts from the committed baselines.
| Accepted pairs | Remaining | Retired | Dismissed |
|---|---|---|---|
| Functions | 0 | 39 | 1 |
| Models | Not initialized | — | — |
Accepted backlog · Weekly audit
Retired means a side was removed, or (functions only) the judge now finds the pair separate. It is not a count of refactors. Dismissals stay separate; an uninitialized model baseline is unknown.