[P1] Add durable background jobs, scheduling, retries, and idempotency

Summary

Provide a durable job profile for work that must outlive an HTTP request: enqueue, lease, execute, retry, compensate, schedule, cancel, and inspect with exact idempotency and uncertainty semantics.

Why this matters competitively

NTNT and orchestration systems expose jobs and workflows. Semaprax can differentiate through typed payloads, stable job/handler identities, explicit capabilities, consumed leases, checkpoint/evidence roots, and honest uncertain-delivery handling.

Current state at the audit baseline

Agent runtimes already model durable checkpoints, uncertain effects, retries, and migration in bounded form.
No general application job queue/scheduler profile is supported.
Database and application profiles can provide persistence and execution hosts.
Agent checkpoints must not be copied blindly into application jobs; shared primitives should be factored explicitly.

Audit baseline: ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (2026-09-11). Recheck main, this issue, and the owning specifications before changing code; later commits may have superseded individual details.

Required outcome

Define typed durable jobs with stable handler and payload schema identities, explicit enqueue/execution capabilities, bounded retry/backoff, leases, idempotency keys, scheduled time, cancellation, dead-letter outcomes, and replayable evidence.

In scope

Job definition and payload schema.
Enqueue and unique/idempotency semantics.
Lease/claim/heartbeat/expiry.
At-least-once baseline with explicit idempotent/transactional patterns.
Retry/backoff and dead-letter state.
Scheduled and recurring jobs.
Cancellation and compensation hooks.
Database-backed reference adapter and deterministic fixture.

Explicitly out of scope

Claiming exactly-once external effects.
Automatic retry of uncertain non-idempotent operations.
Unbounded retries or schedules.
Embedding arbitrary source code in job payloads.
Letting a job receipt authorize publication or payment.

Starting points in the repository

Agent checkpoint/migration/retry implementations
future database profile
std/agent policy helpers
structured task model
new application job package/runtime

Use rg to find the current owners of the named schema, diagnostic, operation, or test before editing. Do not assume a path is still authoritative merely because it is listed here.

Implementation sequence

Specify job states and legal transitions: pending, scheduled, leased, running, succeeded, retryable failure, permanent failure, cancelled, uncertain, dead-lettered.
Derive payload codec/schema from checked Semaprax types and bind every record to job definition ID, handler ID, schema version, enqueue revision, and deployment policy.
Implement enqueue with explicit idempotency key semantics and transaction integration. Define whether duplicate enqueue returns existing job or a closed conflict.
Implement leases as affine grants bound to worker, job generation, attempt, deadline, and handler revision. Expired/stale leases cannot complete jobs.
Define retry policy as pure bounded data; classify failures and uncertain effects. Reserve attempt/cost budgets before execution.
Implement scheduled/recurring jobs with explicit clock/time-zone semantics and catch-up policy.
Reuse generic checkpoint/evidence primitives where semantics match, but create separate application-job schemas and authority boundaries.
Build deterministic in-memory and database-backed adapters and test worker crash/recovery, concurrent claims, network loss, schema migration, and handler revision changes.

Failure and security cases that must be considered

Lease expiry while a worker continues can produce concurrent execution.
Idempotency keys scoped too broadly/narrowly can merge unrelated work or duplicate work.
Cron/time-zone/DST semantics can skip or duplicate schedules.
Retry after uncertain external effect can duplicate side effects.
Payload schema changes can make queued jobs undecodable.

Required tests and evidence

Concurrent enqueue and idempotency races.
Concurrent lease, expiry, stale completion, heartbeat, worker crash, and recovery.
Retryable/permanent/uncertain/cancelled failures and exact attempt ceilings.
Scheduled jobs across clock jumps, DST, missed windows, and recurrence bounds.
Payload/handler revision migration and refusal.
Database transaction integration for enqueue plus application state change.
No execution occurs without a current valid lease and required capabilities.

Acceptance criteria

A reference service can enqueue and execute durable typed jobs.
Delivery semantics and uncertainty are honest and documented.
Retries, schedules, and leases are bounded and deterministic.
Job evidence is replayable but grants no execution authority.
Queued jobs remain migration-safe or fail closed across code changes.

Dependencies and sequencing

[P1] Add typed SQLite and PostgreSQL access with transactions and migrations

Rules for the implementing agent

Read AGENTS.md, docs/COMPLETION-MATRIX.md, docs/ROADMAP.md, docs/QUALITY-GATES.md, and every owning versioned specification named above before editing.
Prefer extending the existing semantic object, operation table, diagnostic catalogue, profile dispatcher, or evidence harness. Do not create a parallel source of truth.
Preserve canonical source, persistent identities, revision binding, fail-closed replay, deterministic rendering, and the separation between evidence and authority.
A local test is local evidence. A hosted run is hosted evidence. A generated artifact is not a support or publication decision. Record exactly what ran and do not widen claims.
Any new wire format, ABI, descriptor, carrier, evidence document, or compatibility surface requires an explicit versioned schema, independent decoder/replay, hostile-input tests, bounds, and nonclaims.
Never grant ambient filesystem, process, network, model, signing, payment, commit, publication, or registry authority merely because a declaration, receipt, test, or proof exists.
Keep changes reviewable. Split implementation into commits that each preserve the existing test suite, and explain any known-answer update field by field.
Do not weaken or delete a failing regression to obtain green CI. Fix the cause or narrow the admitted feature with a stable diagnostic.

Definition of done

The implementation, tests, documentation, status wording, diagnostics, and evidence agree on one exact scope. Targeted tests pass, the applicable repository quality gate passes, hostile cases fail with the documented stable reason, and no unrelated public surface or authority is widened.

feat(jobs): add durable-job lifecycle, lease, retry, and uncertainty …

chore(std): register std/jobs in the standard-library catalog (#192)

The decision procedures landed and execute. The half this issue is named for — surfacing delivery uncertainty to a SEMAPRAX caller — is blocked on #228, with the probe recorded. Landed in 42ae4c4e + a7a57e11.

The #228 boundary, stated precisely

Delivered: the full decision procedure for what an UNCERTAIN outcome means and how it resolves, given that a kind == uncertain was already observed — auto-retry only when the job is idempotent and under its ceiling, otherwise it stays Uncertain pending explicit reconciliation.

Blocked: a checked SEMAPRAX-authored job handler cannot itself produce that observation. Every fallible host op aborts its invocation on failure rather than returning a value — confirmed directly against docs/BOUNDED-LANGUAGE-NETWORK-IO-V1.md: "a nonzero status aborts the enclosing function … exactly like any other fallible host-command operation." Only a Rust-side runner can supply kind == uncertain today, mirroring DatabaseFixture::connection_lost's existing precedent.

No sentinel was invented to paper over this, and the verifier was not touched. The exact follow-on probe is written into docs/DURABLE-JOBS-V1.md#the-228-boundary-what-blocks-a-checked-semaprax-caller.

What shipped

std/jobs — 29 pure functions under useful-data.v1: a ten-state lifecycle (including a non-terminal UNCERTAIN), affine lease legality (claim / begin-execution / heartbeat / completion), bounded retry classification with capped exponential backoff, idempotent enqueue (fresh / duplicate / conflict), tick-clock scheduling with bounded skip-missed catch-up and bounded recurrence, cancellation and compensation, handler/schema-revision refusal, and attempt/cost budget legality.

src/job_fixture.rs — a Rust-only store composing those with a real DatabaseFixture transaction for enqueue, so the ledger row and idempotency check commit atomically. That is the database-transaction integration the issue asks for.

Criterion-by-criterion

Criterion	State
Reference service enqueues and executes durable typed jobs	Partial — proven via the local in-memory fixture; no real service, broker or transport
Delivery semantics and uncertainty honest and documented	Met for the decision-procedure half; the blocked half is documented precisely
Retries, schedules and leases bounded and deterministic	Met
Job evidence replayable but grants no execution authority	Not met — no checkpoint/evidence-root format was composed for jobs specifically
Queued jobs migration-safe or fail closed across code changes	Met — `revision.is_known` / `requires_refusal`, exercised in the fixture

Limitations, stated rather than implied

The concurrency (claim-race) test is a deterministic single-threaded simulation, exactly as database_fixture.rs already does — not proof about a real multi-threaded runner. Scheduling is over an explicit usize tick clock, not wall-clock or calendar time; a real adapter must convert. No broker, durable transport or background scheduler thread exists or should be inferred — this is proof data a runner must obey, not a runner. JobStore is an explicit value a caller must hold, so there is no ambient access, but that is thinner than the first-class capability grant the issue's phrasing suggests.

One composition limit worth recording: std.jobs cannot cross-import std.db from its own .spx source, because std.db is not yet in the compiler's bundled dependency registry (src/project/standard_dependencies.rs). It composes with std.bytes instead and restates two small bounds locally.

Catalog registration is its own commit (a7a57e11), regenerated with the repo's own tool. I verified at integration that it is purely additive — zero deletions — and that std.db's entry from #190 survives intact, so the two package additions from this tranche do not collide.

Gates on the merged tree: job_fixture 13 passed; semaprax check/test/run std/jobs all clean on the interpreter; committed_catalogs_match_the_sources and the std-identity contract test 2 passed; documentation 29; module_size 1; fmt --check and clippy --lib/--bins clean. The full multi-backend standard_library:: gate was deliberately not run — it is one of the >10GB runs.

test(std): execute std.db and std.jobs conformance on native C11 and …

feat(jobs): add a replayable job evidence log, closing #192's authori…

style: apply cargo fmt to the job evidence log

Update: the "job evidence replayable but grants no execution authority" gap noted Not met in the prior comment is closed on main (35b608c8 + 4bd32942 + ca6ad292).

src/job_evidence.rs adds JobEvidenceLog: an append-only, domain-separated SHA-256 hash chain over typed entries mirroring every JobStore transition. JobEvidenceLog::replay independently recomputes a job's final lifecycle state from the recorded entries alone (reusing job_fixture::decisions, not a second parallel state machine) and returns FinalStateMismatch when a recorded input's recomputed outcome disagrees with the log's own claimed final state. Appending, hashing, and replaying are inert data operations — no socket, thread, lease claim, or execution — matching AGENTS.md's "a settlement or concurrency model is proof data, not permission to perform a physical finalizer, spawn runtime work, or publish an artifact."

Explicitly documented, not swept under the rug: Claimed's is_due and Completed's outcome_kind are asserted facts this format does not itself check against a clock, so replay only proves internal legality and agreement with the log's own claim, not that the asserted facts were true. A terminal state (SUCCEEDED/PERMANENT_FAILURE/CANCELLED/DEAD_LETTERED) can never be reopened by a later entry, mirroring this repository's sticky-failure-selection invariant.

Updated criterion table

Criterion	State
Reference service enqueues and executes durable typed jobs	Partial (unchanged) — the in-memory fixture enqueues, leases, and records handler outcomes; it does not invoke a handler itself, run a broker, or hold any execution authority, by design
Delivery semantics and uncertainty honest and documented	Met
Retries, schedules and leases bounded and deterministic	Met
Job evidence replayable but grants no execution authority	Met — `src/job_evidence.rs`, above
Queued jobs migration-safe or fail closed across code changes	Met

Verification this session (local evidence only — nothing here is hosted/CI-verified)

Re-ran the full gate set against the current merged tree (HEAD 288b4a77, same as origin/main):

cargo test --locked -p semaprax --lib job_fixture:: — 13 passed, exit 0
cargo test --locked -p semaprax --lib job_evidence:: — 10 passed, exit 0
cargo test --locked --test documentation — 29 passed / 1 ignored, exit 0
cargo test --locked --test module_size — 1 passed, exit 0
cargo fmt --all --check — exit 0

(cargo test --test project -- standard_library:: was attempted but externally SIGTERM'd mid-run on a resource-saturated shared host after every visible sub-test had passed and none had failed; not re-run given host load, and not one of this session's required gates.)

No further code change from this session — an independent pass at this same gap (working blind, before discovering it was already merged) converged on a byte-identical src/job_evidence.rs to what's already on main, which is itself a form of cross-check. Leaving this issue open given the one remaining, architecturally-intentional Partial.

test(jobs): reject an evidence-log claim asserted before a scheduled …

Audited against main. issue-192-durable-jobs is an ancestor of origin/main — it ships as std/jobs, src/job_fixture.rs, src/job_evidence.rs + docs/DURABLE-JOBS-V1.md. Not closing: one acceptance criterion is blocked on a language gap.

Criterion	Status	Evidence
Reference service enqueues and executes typed durable jobs	met at fixture level	`JobStore` composes enqueue/claim/heartbeat/complete/cancel over `DatabaseFixture`
Delivery semantics and uncertainty honest and documented	met	`UNCERTAIN` (state 8) is a sticky resting state — never silently retried or dead-lettered
Retries, schedules and leases bounded and deterministic	met	`retry.backoff_ticks` `ensures result <= max_ticks`; `schedule.catch_up_next_run` bounded by `max_catch_up`; leases are affine `(lease_generation, job_generation, deadline_tick)` grants
Job evidence replayable, grants no execution authority	met	`JobEvidenceLog::replay` independently recomputes final state from inert data
Queued jobs migration-safe or fail closed	met	`revision.is_known`/`requires_refusal`, via `revision_check_refuses_an_unknown_bound_revision`
Surface uncertain outcomes to checked SEMAPRAX handlers	not met — blocked on #228	every fallible host op aborts the invocation rather than returning an inspectable three-way outcome
Real message broker or durable transport	not shipped	in-memory fixture, by design
True concurrency rather than simulated	not proven	the concurrent-claim test is a deterministic single-threaded simulation, and is disclosed as one

Added this pass (merge cce695a2): a_scheduled_job_claimed_before_it_is_due_is_rejected in src/job_evidence.rs. replay() already recomputed claim_is_legal for every Claimed entry, but no test proved that a SCHEDULED job asserted-claimed with is_due: false is rejected at the evidence-replay layer — that case existed only in the pure .spx decision procedure (std.jobs.tests.claim_legality). The replay layer is the one a real deployment verifies submitted job histories against, so the gap mattered. It asserts IllegalTransition { entry_index: 1 } specifically. No behaviour changed; this closes coverage, it does not fix a bug.

Test-name honesty check — clean. Every test name in job_fixture.rs, job_evidence.rs and std/jobs/src/tests.spx was checked against what it exercises. The concurrency simulation is labelled as a simulation rather than as proof of concurrency, which is the honest form.

Caller check: job_evidence/job_fixture are called by src/lib.rs, each other, and their own tests. std.jobs is called by db_jobs_backend_audit.rs and execution_matrix.rs. No runtime caller outside tests.

Evidence class: local re-runnable. Nothing hosted.

Remaining for closure: #228 — until a checked handler can distinguish "definitely did not happen" from "may have happened and I cannot tell", the honest-uncertainty criterion cannot be met. A lane is working #228 now.

fix(jobs): close the completion-durability gap in the #192 job fixture

Progress in eec9b187. Not closing — one criterion remains genuinely blocked.

The gap I found and closed. complete() only ever mutated the in-memory job record. The ledger's state column, written once as a placeholder by enqueue, was never rewritten by anything — so nothing in the system distinguished "the handler ran and the outcome is durably recorded" from "the handler ran and the worker crashed before that record committed." That is precisely the second of the three idempotency cases that are usually faked, and it was open here.

complete_durable commits the ledger write before advancing in-memory state, and when the commit cannot be confirmed it forces the existing Uncertain resting state rather than the handler's reported outcome. complete() itself is unchanged. Supporting primitive: DatabaseFixture::update_column, added alongside the existing insert/select_eq.

Proved by breaking it, not by assertion:

Deleting the commit_confirmed gate so the method unconditionally applies the outcome → complete_durable_forces_uncertain_rather_than_the_handlers_outcome_when_the_ledger_commit_is_never_confirmed failed with left: Succeeded, right: Uncertain, every other test still passing. Restored.
Relaxing update_column's shape check to accept any type → update_column_rewrites_matching_rows_and_refuses_a_shape_mismatch failed with left: Ok(1), right: Err(RowShapeMismatch). Restored.

Which idempotency cases the tests genuinely reach, stated honestly:

Completed but not durably recorded before a crash — reached, via a stuck-open ledger transaction as a deterministic proxy for a crash.
The same key submitted twice concurrently — not reached. The existing concurrent-claim test is a disclosed single-threaded simulation and I did not change that.
Retry after partial progress — not covered beyond what attempt/backoff already model.

Reachability, unchanged and stated plainly: job_fixture/job_evidence are Rust-only and called by lib.rs, each other, and their own tests. std.jobs is called from the backend audits and the scaffold template. There is no runtime caller outside tests — no broker, no scheduler thread, nothing claiming jobs on a timer. That is deliberate, per the invariant that a settlement or concurrency model is proof data, not permission to spawn runtime work; but it means "durable jobs" here means a checked decision procedure plus a fixture, not a running system.

Still NOT met, and deliberately not worked around: surfacing an uncertain outcome to a checked SEMAPRAX handler. Every fallible host op aborts the invocation rather than returning a value, so a handler cannot observe the UNCERTAIN state that the decision procedure models. That is #228, whose ABI change sits behind the SPX-AI-019..025 review checkpoint. Four independent sessions have now reached this same conclusion; inventing a status vocabulary here to close the criterion would be the wrong kind of progress.

Evidence: --lib fixture:: → 39 passed, 0 failed (14 database_fixture::, 15 job_fixture::, 10 unrelated pre-existing matches). Local, offline, re-runnable; not hosted.

Separately noted: --test module_size is red on main for src/cli_driver.rs, src/hir/validation.rs and tests/project/standard_library.rs, none of which this touches. Being fixed by extraction rather than by raising the recorded budgets.

fix(jobs): close the completion-durability gap in the #192 job fixture

feat(jobs): commit a job's completion to the ledger before advancing …

Pushed to main in 4103e23f1471fd0689c9dca01d147e42d3a53e85.

Added an explicit host-driven single-job checkpoint runtime over JobStore and JobEvidence. It admits payloads using a compiled interaction schema, persists original attempt times for replay, uses CAS generations with native advisory file locking and same-directory replacement, recovers Running as Uncertain, and poisons runtime instances after failed checkpoint writes.

The host handler seam is not proof of a compiled callable binding. DatabaseFixture remains an in-memory decision mirror; this is not a physical database transaction with application state. Source-handler/deployment identity binding, general queue/service integration, runtime recurrence/heartbeat/cancellation, and physical database integration remain open. The file store makes no power-loss or hostile-root guarantee. Keeping this issue open.

Local evidence: 34 job fixture/evidence/runtime tests, including compiled-schema retry/recovery and checkpoint-failure poisoning. The combined library/binary compile passed. No full quality profile or hosted CI run was performed for this batch, per the requested implementation-first workflow.

Implemented persistent cancellation and recurring execution lifecycle controls (commit cbce6ed):

Runtime cancellation uses the existing reducer and immediately checkpoints; persistence failure poisons the live runtime.
Successful recurring executions checkpoint exact next due tick and occurrence count, with additive RecurringAdvanced evidence tag 9. Recovery independently replays the recorded schedule.
Checked schedule arithmetic is preflighted before claim or handler dispatch. Recurrence cannot revive cancelled/running jobs, and each new occurrence gets a fresh attempt budget.
Existing checkpoint envelope and earlier evidence tags remain compatible.

Focused cancellation/recovery, recurring/recovery, overflow/no-effects and hostile evidence tests passed. No full or hosted CI gates run.

Issue remains open for source handler bindings, broader queue/heartbeat execution, production database/transaction integration and full reference-app acceptance. These changes extend the real host-driven single-job runtime, without claiming autonomous execution or physical database atomicity.

Closure audit @ e9ab9242 — NOT CLOSEABLE.

The lease-fencing defect found here is fixed (52fa3980): lease generations now derive from a monotonic lease_epoch rather than the outstanding lease, so a reclaimed worker's stale completion can never land over the new holder's run.

Blocker is this issue's own listed dependency: no physical SQLite/Postgres adapter exists — only an in-memory DatabaseFixture. True multi-process concurrency is therefore unproven, which is the property this issue exists to establish.

Correction: a claim I made about this issue today was wrong.

An audit I ran earlier reported that it had grepped this repository for any durable-job or lease primitive and found none, and I passed that on. It is false. I verified against the tree at 81031817:

src/job_fixture.rs (1448 lines) — JobStore over DatabaseFixture: ledger-transactional enqueue, lease claim/heartbeat/expiry with epoch-fenced generations, retry/backoff/dead-letter, idempotent completion.
src/job_evidence.rs (809 lines) — JobEvidenceLog: append-only domain-separated SHA-256 hash chain with an independent replay() that recomputes final state.
src/job_runtime.rs and src/job_runtime/{source_handler,source_handler_tests,tests}.rs — host-driven checkpoint runtime, versioned checkpoint wire format with CAS-generation file locking, heartbeat and lease extension, source-handler binding with deployment digest.
std/jobs/src/jobs.spx — 29 pure functions: ten-state lifecycle, affine lease legality, bounded retry classification, idempotent enqueue, tick-clock scheduling with bounded catch-up.
docs/DURABLE-JOBS-V1.md (519 lines).

Registered at src/lib.rs:210-212. Gate, run locally just now:

cargo test --locked -p semaprax --lib -- job_fixture:: job_evidence:: job_runtime::
test result: ok. 51 passed; 0 failed; 0 ignored; 3743 filtered out

That count already includes the hostile coverage a greenfield build would have been chasing: checkpoint_decoder_refuses_trailing_or_noncanonical_boolean_bytes, recover_refuses_a_checkpoint_whose_schema_digest_no_longer_matches, physical_store_refuses_stale_cas_and_retains_current_generation, a_recovered_succeeded_job_never_redispatches_its_handler, retryable_failures_back_off_and_dead_letter_at_the_ceiling, scheduled_job_claims_only_once_due_and_recurs_with_bounded_catch_up, and a_reclaimed_workers_stale_completion_can_never_land_over_the_new_holders_run — the last covering a real lease-fencing bug this codebase already found and fixed in 52fa3980.

No code was written. The agent I dispatched checked the premise before building, found the existing implementation, and refused the task rather than standing up a second, less mature set of primitives beside it. That was the correct call and I'm recording it here so the mistake doesn't propagate further.

The residual recorded in this issue's own last closure audit stands unchanged and is accurate: a real SQL adapter behind DatabaseFixture (still in-memory), true multi-process concurrency (still a disclosed single-threaded simulation), and the acceptance criterion blocked on #228, where a checked .spx handler cannot observe an UNCERTAIN host outcome because every fallible host op aborts rather than returning a value.

Worth noting for anyone auditing this backlog: today's errors have run in both directions. Four issues understated what was built; this one claimed nothing was built when a great deal was. Verify against the tree, not against the record — including mine.

test(jobs): race the claim and CAS paths on real OS threads

A correction to how this issue has been classified, because it changes what unblocks it.

An audit sweep listed #192 as offline-actionable with two remaining gaps: "replace the in-memory DatabaseFixture with a real SQL adapter" and "implement true multi-process concurrency (currently a disclosed single-threaded simulation)". Checked against the code, both are one gap, and it is not offline-actionable.

On the concurrency half — the existing disclosure is already accurate, and better than the sweep implied. src/job_fixture.rs:10-26 describes exactly what it has: "a deterministic single-threaded scripted concurrent-claim simulation, a second concurrent-claim race driven by genuine OS threads racing under a Barrier (proving the same fencing invariant under real nondeterministic scheduling rather than a call order this module's own test chose)". d6a12309 landed that OS-thread race. Nothing here is passing off a simulation as real concurrency.

Multi-process is not an increment on that; it is blocked by the storage question. DatabaseFixture is BTreeMap<String, Table> in process memory (src/database_fixture.rs:166-171). Separate processes share no address space, so there is nothing for a second process to contend over — a multi-process claim race is only meaningful once the store is a shared durable medium. The two "gaps" are therefore the same gap.

And that gap needs a decision, not an afternoon. This repository has no SQL dependency at all — no rusqlite, no libsqlite3-sys, no postgres in Cargo.toml. Adding one means a new C dependency, a Cargo.lock change, and passage through the supply-chain release-blocker job. AGENTS.md also forbids build-time network access, so any such dependency has to be vendored or pinned accordingly. That is a maintainer call about the project's dependency surface, not something an implementation pass should decide unilaterally — which is presumably why every prior pass stopped where it did.

Suggested reclassification: blocked on a storage-dependency decision. Two things would unblock it, and they are quite different in cost:

Approve a specific embedded SQL dependency (and its vendoring story), after which the adapter and a genuine multi-process claim race both become ordinary work; or
Decide the durable medium is something already present — the atomic-write provider route this module already uses for classified uncertainty (docs/HOST-OPERATION-OUTCOME-V1.md) — in which case multi-process contention can be built on that without any new dependency, and the "real SQL adapter" requirement should be rewritten to say so.

Option 2 may well be the smaller and more honest path, since it keeps the capability surface unchanged. Either way the choice belongs to a maintainer. Nothing in this comment changes code.

test(jobs): race enqueue idempotency under genuine OS-thread concurrency

Re-audit at b9540a2e — one genuine gap found and closed; the rest was already shipped.

Audited src/job_fixture.rs, src/job_runtime.rs (+ source_handler), src/job_evidence.rs and std/jobs/src/jobs.spx against every acceptance criterion and every entry on the required-tests list. Nearly all of it was already in place:

A reference service enqueues and executes durable typed jobs — JobRuntime::enqueue/drive_once, with source_handler.rs driving a real checked SEMAPRAX callable to completion.
Delivery uncertainty is honest and documented — an UNCERTAIN state, uncertain.retry_is_permitted, and complete_durable forcing Uncertain when a ledger commit is unconfirmed.
Retries, schedules and leases are bounded and deterministic — capped retry_backoff_ticks, schedule_catch_up_next_run bounded by max_catch_up, and lease_epoch fencing.
Job evidence is replayable with no execution authority — an append-only SHA-256 hash chain with independent replay and tamper detection.
Queued jobs are migration-safe or fail closed — revision_check plus a checkpoint refusal on schema-digest drift.

The one real gap: "concurrent enqueue and idempotency races" is its own line item on the required-tests list, and it had no concurrent test. Every existing idempotency test drove JobStore::enqueue's find-then-insert sequence sequentially. Only claim had a genuine race test. A find-then-insert pair is exactly the shape that passes every sequential test and still admits two winners under contention.

Closed in 486a14ee with two tests using real OS threads, barrier-synchronised, 16 workers × 25 iterations, following the existing claim-race test's pattern:

real_os_thread_concurrent_enqueue_race_creates_exactly_one_job_for_the_same_key — same key, same descriptor. Exactly one Created, every other call a Duplicate of it, one ledger row. Catches a find-then-insert race outside the lock, or any bug letting two threads both pass the uniqueness check.
real_os_thread_concurrent_enqueue_race_with_differing_descriptors_conflicts_the_losers — same key, half the workers using a different descriptor. Whichever wins, matching calls must be Duplicate and non-matching calls must be Conflict. Catches a silent merge onto the winning row, or a spurious second Created.

Evidence: --lib job_fixture:: → 20 passed, --lib job_runtime:: → 22 passed, --lib job_evidence:: → 13 passed, 0 failed throughout. All local.

Residual scope, honestly:

No physical database driver behind DatabaseFixture. The transaction integration for enqueue is proven against the in-memory fixture only; a real Postgres/SQLite driver remains separate work.
Multi-process concurrency (separate address spaces) is untested. Only real-OS-thread concurrency within one process is proven, as the doc's non-claims already disclose.
DST and wall-clock scheduling are out of reach: the profile takes an explicit usize tick, and the language has no calendar library or numeric cast to build a wall-clock adapter on. This is a documented scope boundary, not a code gap, and was not attempted.

Refs, not Closes — those three non-claims mean not every criterion is unconditionally met, though the large majority were already shipped before this pass.

Gotcha worth recording for gate scripts: cargo test <filter_a> <filter_b> with two positional filters is rejected by cargo itself ("unexpected argument"); the two suites must be separate invocations.

feat(job-service): reference project that enqueues and executes a rea…

A reference service that genuinely executes durable jobs from .spx source (fb5f1de0) — this issue's first acceptance criterion.

examples/job-service-project. job_service.core composes bundled std.jobs into a claim/idempotency/retry/dead-letter decision walk, gated on interpreter, native C11 -O0/-O2, and Core Wasm under Node. job_service.app declares a real checked handler, handle_report_job, with its ReportJob payload record.

It executes, rather than models. tests/useful_data/job_service_project.rs binds job_service.app.handle_report_job — compiled from the project's own retained .spx source, not a Rust stand-in — through SourceJobHandlerBinding/drive_checked_source_job, enqueues a real JobRuntime job, and drives it to completion through the retained-call interpreter seam. JobRuntime owns every checkpoint, lease, retry, backoff and dead-letter transition; the .spx handler owns only effect-free classification of an already-admitted payload. That division is the point: the handler gains no authority by being callable.

Four previously undocumented constraints surfaced, now written up in the README — each of which silently breaks a binding that otherwise looks correct:

SourceJobHandlerBinding re-verifies its bound file standalone, so that file must have zero cross-module use and its own fn main.
Only a manifest's entry or its single tests module may declare main at all (SPX-G172).
The bound file must be exactly comment-free. graph::revision hashes comment-dropped canonical text while the project's per-file source_revision hashes raw bytes including comments, so a single // causes a SourceRevisionMismatch on a file that passes semaprax fmt --check. This is the same comment-versus-canonical-bytes seam rename_display_name preview is unsatisfiable for any project with commented bundled dependencies (SPX-G525) #274 found behind SPX-G525, surfacing in a second place.
Ordinary interpreter entry/test execution refuses any call reaching a record-parameter function (SPX-F102), which is why main() stays a trivial 0.

Budget: measured with builder_bytes_breakdown at 12,616,649 bytes — 67% of the 18,874,368 cap, roughly a third headroom — so a service with a minimal dependency closure fits comfortably. That is a useful counterweight to the 92.7% figure measured on task-service-project: the ceiling is reachable, but not inherent to a reference service.

Four fault-injection cycles, each confirmed red then reverted: a wrong idempotency expectation breaks the test aggregator; a constant-returning handler turns Succeeded into PermanentFailure; max_attempts=1 collapses Scheduled → DeadLettered into an immediate dead-letter; a mismatched payload descriptor turns Duplicate into Conflict.

Disclosure. A concurrent lane's yield rollout left the crate non-compiling for this lane's entire session, so its verification ran via the pre-built target/debug/semaprax binary and rustc --test against a pre-built rlib — 6 passed, run four times with consistent results. The crate compiles again as of 24c1d166, and I re-ran the suites centrally before pushing.

Residual: docs/COMPLETION-MATRIX.md not updated (shared file, other lanes active). No granular per-module builder_bytes breakdown of the final three-file layout — semaprax check succeeding is authoritative that it fits, but the per-module numbers are for the earlier four-file shape. No std/ source changed.

Refs, not Closes — the DST/wall-clock and multi-process non-claims stand.

Re-audit at 12373af8 — still NOT closeable; independently re-verified this pass rather than quoted from this thread's history.

Fresh runs:

cargo test --locked -p semaprax --lib -- job_fixture:: job_runtime:: job_evidence::
test result: ok. 55 passed; 0 failed; 4084 filtered out

cargo test --locked --test useful_data job_service_project::
test result: ok. 6 passed; 0 failed; 167 filtered out

Five acceptance criteria are MET with executed evidence: durable typed enqueue/execute (entry_and_conformance_return_zero_on_interpreter_native_and_wasm), honest delivery-uncertainty documentation (uncertain_outcome_stays_uncertain_for_a_non_idempotent_handler_until_reconciled), bounded retries/leases (retryable_failures_back_off_and_dead_letter_at_the_ceiling, heartbeat_extends_the_deadline_and_is_refused_once_stale), replayable evidence with no execution authority (10 replay/tamper tests), and fail-closed migration (revision_check_refuses_an_unknown_bound_revision).

Two required tests remain genuinely unmet, and neither is a small fix:

Database-backed enqueue + app state. src/database_fixture.rs:166-168 is still an in-memory BTreeMap; Cargo.toml has no rusqlite/libsqlite3/postgres/sqlx — re-confirmed this pass. This issue's own dependency [P1] Add typed SQLite and PostgreSQL access with transactions and migrations #190 is closed, but [P1] Add typed SQLite and PostgreSQL access with transactions and migrations #190's final comment records the driver decision as still HUMAN_BLOCKED, so it closed without shipping a physical driver. Adding one is a new C dependency plus a supply-chain and vendoring decision reserved for the maintainer. Multi-process concurrency is the same root cause — there is no shared durable medium for separate processes to contend over.
DST / wall-clock scheduling. Tick-based catch-up and missed windows are covered (scheduled_job_claims_only_once_due_and_recurs_with_bounded_catch_up; schedule_catch_up_next_run(999,100,10,2) == 130), but SEMAPRAX has no calendar/timezone library, so DST semantics cannot be built. This is a disclosed non-claim, not a defect.

One detail worth recording, since #228 closing could be read as having settled it: src/job_runtime/source_handler.rs:336-348 (map_outcome) maps a checked .spx handler's 0/1/2 to Succeeded/RetryableFailure/PermanentFailure, and has no branch producing HostJobOutcome::Uncertain — only the Rust host adapter can reach that state. This is deliberate and documented in place: "The checked callable is effect-free. A language or capacity failure therefore cannot report an external delivery result; classify it as terminal rather than inventing a source-visible uncertainty value." So the profile's uncertainty documentation is accurate, but the "surface UNCERTAIN to a checked handler" ambition is still undelivered even though its language blocker (#228) is gone, and that residual is not currently restated in this issue's criteria table.

No code changed this pass. Forcing either gap would mean adding an undeclared dependency or weakening the effect-free handler boundary — both prohibited here. Remaining work needs a maintainer decision on the storage dependency: vendor a SQL driver, or repurpose the existing atomic-write provider as the durable medium.

docs(decisions): record the signing and durable-job storage decisions

Decision recorded — docs/decisions/0005-durable-job-storage-medium.md (commit 94924c98). The storage dependency this issue was blocked on is now chosen.

Durable jobs run on this repository's own atomic-generation substrate, behind a sealed JobStore adapter seam. No SQL driver in the default build. A vendored SQLite adapter stays possible and additive behind that seam, off by default, and is not required to close this issue.

Why not a bundled SQL driver:

It contradicts the capability invariant. rusqlite with bundled compiles vendored C and needs unsafe FFI, in a workspace where eight crates declare unsafe_code = "forbid". PostgreSQL is worse here — a running server and network authority the compiler is forbidden to acquire ambiently.
It expands the supply chain and the build matrix (C toolchain on Linux, macOS, Windows MSVC) to obtain a property this repo can already express.
SQLite's durability defaults mislead: commits are not durable under default settings, and synchronous=NORMAL in WAL mode permits losing recently acknowledged writes after OS or power failure. Taking the dependency and still having to reason carefully about fsync buys less than it looks like.

Why the existing substrate is right: ADR 0002 already publishes one complete immutable generation through ACTIVE (.semaprax-workspace/ACTIVE) — staged content pivoted by a single atomic rename. That is the canonical durable-commit pattern, already crash-safe, already deterministic, already load-bearing here. Durable jobs built on it inherit deterministic replay and "failed transactions leave authoritative state unchanged" for free. It also supplies the shared medium that multi-process concurrency testing currently lacks — so that second gap closes with the same work, since both had one root cause.

The honest caveat, recorded in the ADR so it isn't rediscovered: append-only plus atomic rename is crash-friendly but not automatically durable. Acknowledging a job as durably accepted requires write → flush → fsync the file → rename on the same filesystem → sync the directory entry, and only then acknowledge; group-commit keeps that affordable. Whichever durability level is chosen must be stated in the profile documentation, not implied — a store that has not fsynced has not committed and must not report that it has.

The competitive argument: "durable background jobs, scheduling, retries and idempotency, with no database to run" is genuinely differentiating — the same move SQLite made one level down. Every comparable runtime makes you provision Postgres or Redis before a job can survive a restart. It also keeps the --locked --offline CI story intact: no service, no container, no credentials.

One scoping consequence that needs your eye: this issue's required test says "database transaction integration for enqueue + app state." Read against the JobStore seam, that criterion is satisfiable without SQL. If the intent was specifically a SQL engine, this ADR re-scopes the criterion rather than satisfying it — flagging that explicitly rather than quietly assuming the looser reading.

Still genuinely out of scope regardless: DST/wall-clock scheduling, since the language has no calendar library — a disclosed non-claim, not a defect.

docs(summary): list ADR 0004 and ADR 0005

feat(jobs): add a file-backed JobStore seam on the atomic-generation …

Closing — with one criterion explicitly re-scoped, not silently checked off

Most of #192 shipped before this session (std/jobs, src/job_fixture.rs, src/job_evidence.rs, src/job_runtime.rs, examples/job-service-project, docs/DURABLE-JOBS-V1.md). What was missing was a durable-job persistence seam. da68ff46 adds it: src/durable_jobs/ — model, retry, codec, durable_fs, store — 1994 lines, every file under the size cap, one line in src/lib.rs, nothing else touched. No new SPX- code, so nothing to regenerate.

The one criterion that does not close as written

"Database transaction integration for enqueue plus application state change."

Read as transactional integration with the durable store — atomicity, no torn writes, no double execution — it is met: enqueue_with_side_record_commits_both_or_neither_on_success and a_fault_during_the_joint_commit_leaves_neither_the_job_nor_the_side_record_visible, the latter fault-injected at three points in the durable-write sequence.
Read as a SQL database, literally — it is blocked by an invariant, not by effort. Eight crates in this workspace forbid unsafe, and a Postgres client needs ambient network authority, which AGENTS.md prohibits outright. No amount of work unblocks that; it would take a decision to relax a capability invariant.

ADR 0005 records that re-scope and the reasoning, and was written precisely because this criterion was ambiguous. Closing on the first reading, with the second named here rather than quietly dropped. If the literal SQL reading was intended, reopen and it becomes a capability-invariant question rather than an implementation one.

The rest

Requirement	Verdict
Concurrent enqueue and idempotency races	MET — `racing_enqueue_calls_with_the_same_idempotency_key_create_exactly_one_job` (16 OS threads, `Barrier`-synchronized), plus `a_conflicting_payload_under_the_same_key_is_a_closed_refusal_not_a_merge`
Lease, expiry, stale completion, heartbeat, crash, recovery	MET — pre-existing `src/job_fixture/tests.rs`, extended here with generation fencing (`completing_with_a_stale_lease_generation_is_refused_and_never_mutates_the_job`)
Retryable / permanent / uncertain / cancelled, exact attempt ceilings	MET — including `a_retryable_job_that_never_succeeds_is_dead_lettered_at_the_ceiling_not_retried_forever`, so the retry loop has a terminal state
Scheduled jobs across clock jumps, missed windows, recurrence bounds	MET, tick-based. DST is an explicit non-claim — this is a tick clock, not a wall clock, and `docs/DURABLE-JOBS-V1.md` says so
Payload/handler revision migration and refusal	MET — pre-existing
No execution without a current valid lease and capabilities	MET — four refusals including `a_forged_lease_token_naming_the_wrong_worker_is_refused`

Crash durability is proved by dropping the store and reopening from disk (an_acknowledged_job_survives_a_simulated_crash_and_reopen). The durability discipline is the one that is usually got wrong and is right here: durable_fs.rs fsyncs the staging file (:91) and its parent directory (:118), because a file-only fsync can lose the directory entry that names it on crash; the non-portability of directory fsync outside Unix is documented rather than ignored.

Determinism holds with no wall clock anywhere in the model — identical_operation_sequences_produce_byte_identical_generations and job_ids_are_assigned_in_strictly_increasing_enqueue_order, with BTreeMap-ordered encoding so no unordered iteration reaches the bytes on disk. A source-scan regression (module_source_never_reaches_for_network_process_or_env_authority) keeps the module free of ambient authority.

cargo test --locked -p semaprax --lib durable_jobs → 40/40, with -- --list run first to rule out the zero-match false pass. cargo check --locked -p semaprax --all-targets clean.

Navigation Menu

Repository navigation

[P1] Add durable background jobs, scheduling, retries, and idempotency #192

Summary

Why this matters competitively

Current state at the audit baseline

Required outcome

In scope

Explicitly out of scope

Starting points in the repository

Implementation sequence

Failure and security cases that must be considered

Required tests and evidence

Acceptance criteria

Dependencies and sequencing

Rules for the implementing agent

Definition of done

The #228 boundary, stated precisely

What shipped

Criterion-by-criterion

Limitations, stated rather than implied

Updated criterion table

Verification this session (local evidence only — nothing here is hosted/CI-verified)

5 remaining items

Closing — with one criterion explicitly re-scoped, not silently checked off

The one criterion that does not close as written

The rest

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[P1] Add durable background jobs, scheduling, retries, and idempotency #192

Description

Summary

Why this matters competitively

Current state at the audit baseline

Required outcome

In scope

Explicitly out of scope

Starting points in the repository

Implementation sequence

Failure and security cases that must be considered

Required tests and evidence

Acceptance criteria

Dependencies and sequencing

Rules for the implementing agent

Definition of done

Activity

wsdt commented on Sep 12, 2026

The #228 boundary, stated precisely

What shipped

Criterion-by-criterion

Limitations, stated rather than implied

wsdt commented on Sep 12, 2026

Updated criterion table

Verification this session (local evidence only — nothing here is hosted/CI-verified)

wsdt commented on Sep 12, 2026

wsdt commented on Sep 12, 2026

wsdt commented on Sep 13, 2026

wsdt commented on Sep 13, 2026

5 remaining items

wsdt commented on Sep 18, 2026

wsdt commented on Sep 18, 2026

wsdt commented on Sep 19, 2026

wsdt commented on Sep 19, 2026

wsdt commented on Sep 19, 2026

wsdt commented on Sep 19, 2026

wsdt commented on Sep 19, 2026

wsdt commented on Sep 19, 2026

Closing — with one criterion explicitly re-scoped, not silently checked off

The one criterion that does not close as written

The rest

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions