Repository navigation
[P1] Enforce model call, token, context, latency, retry, failover, and cancellation budgets #179
Description
Summary
Define and enforce the operational policy around live model invocation. Every call, retry, failover, streamed byte, context token, output token, elapsed deadline, and cost reservation must be bounded and reflected in evidence.
Why this matters competitively
Neam exposes budgets and guards; production orchestration products expose routing and retry. Semaprax can outperform them by making these policies deterministic intersections of source ceilings, deployment ceilings, and invocation budgets with explicit uncertainty semantics.
Current state at the audit baseline
- Agent deployments and typed effect runs already intersect several call/byte/turn ceilings.
- Per-operation checkpoints and uncertain-effect handling exist for tool effects.
- Live model provider calls and provider billing are not yet integrated.
- Cancellation cannot be assumed to cancel remote provider work.
Audit baseline: ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (2026-09-11). Recheck main, this issue, and the owning specifications before changing code; later commits may have superseded individual details.
Required outcome
Introduce a provider-neutral model policy that reserves budgets before dispatch, tracks attempted and accepted usage, applies bounded retry/failover only for explicitly safe failure classes, and never treats cancellation or transport failure as proof of non-occurrence.
In scope
- Maximum calls, retries, providers, context bytes/tokens, output bytes/tokens, aggregate tokens, latency/deadline, and estimated cost.
- Source/deployment/invocation ceiling intersection.
- Retryability classification and idempotency/uncertainty rules.
- Provider failover with exact deployment policy.
- Cooperative cancellation and terminal uncertainty.
- Checkpoint behavior before and after model dispatch.
- Usage accounting on malformed or rejected output.
Explicitly out of scope
- Unlimited automatic retry.
- Retrying an uncertain billable call as though nothing happened.
- Using provider-reported token counts as unverified language truth.
- Selecting providers dynamically outside the deployment policy.
- Refunding budgets automatically on provider or decoder failure.
Starting points in the repository
- AgentDeployment and Direct Runtime v2
- AGENT-OPERATION-CHECKPOINT-V2.md
- typed effects budget code
- model.invoke contract
- execution/evidence root modules
Use rg to find the current owners of the named schema, diagnostic, operation, or test before editing. Do not assume a path is still authoritative merely because it is listed here.
Implementation sequence
- Specify
ModelBudget v1,ModelRetryPolicy v1, and a closed terminal/failure classification. Separate requested, reserved, attempted, provider-reported, locally measured, accepted, and reconciled usage. - Compute effective limits as the minimum of source Agent ceilings, deployment policy, and invocation budget. Reject contradictory or zero-impossible policies before handler access.
- Reserve one call and worst-case or configured cost/token capacity before dispatch. Persist intent in durable mode before calling the provider.
- Classify failures into definitely-not-dispatched, rejected-before-processing, completed-with-response, uncertain, and provider-reported retryable. Automatic retry is allowed only for classes explicitly proven safe by the adapter contract.
- Implement bounded backoff as deterministic policy data. Clock waiting is a host effect; policy computation itself stays pure.
- For failover, bind the ordered provider/model alternatives and maximum attempts into DeploymentRoot. Every attempt receives a unique ordinal and evidence record.
- Check cancellation before dispatch, during streaming, and after provider completion. A cancelled wait may yield uncertain remote status and must retain reserved/attempted usage.
- Stop decoding and further attempts when any aggregate ceiling is reached. Record the exact limit that selected the terminal result.
Failure and security cases that must be considered
- Provider timeout can occur after the provider completed and billed the call.
- Retrying structured-output rejection may amplify cost unexpectedly.
- Provider tokenizers differ; local estimates and provider usage must remain distinct.
- Backoff can exceed the invocation deadline.
- Failover can unintentionally expose data to a provider not authorized for the task.
Required tests and evidence
- Exact boundary and maximum-plus-one tests for every budget.
- No dispatch on missing capability, invalid policy, exhausted reservation, stale deployment, or pre-cancellation.
- Retry occurs only for declared safe classes and never for uncertain delivery.
- Failover follows the exact ordered deployment policy and confidentiality classification.
- Cancellation at every boundary produces deterministic local state and honest remote-status uncertainty.
- Malformed/rejected output still charges attempted bytes/tokens/calls as specified.
- Durable replay performs no repeated provider call for recorded completion.
Acceptance criteria
- Every model attempt is bounded before dispatch.
- Retry/failover behavior is deterministic, deployment-bound, and evidence-bearing.
- Cancellation never falsely asserts provider non-occurrence or no billing.
- Usage fields distinguish estimates, local measurements, provider reports, and reconciliation.
- No automatic budget refund or authority widening occurs.
Dependencies and sequencing
- [P0] Add a provider-independent model.invoke effect to the source-native Agent runtime
- [P0] Derive streaming structured-output decoding from the checked Proposal type
Rules for the implementing agent
- Read
AGENTS.md,docs/COMPLETION-MATRIX.md,docs/ROADMAP.md,docs/QUALITY-GATES.md, and every owning versioned specification named above before editing. - Prefer extending the existing semantic object, operation table, diagnostic catalogue, profile dispatcher, or evidence harness. Do not create a parallel source of truth.
- Preserve canonical source, persistent identities, revision binding, fail-closed replay, deterministic rendering, and the separation between evidence and authority.
- A local test is local evidence. A hosted run is hosted evidence. A generated artifact is not a support or publication decision. Record exactly what ran and do not widen claims.
- Any new wire format, ABI, descriptor, carrier, evidence document, or compatibility surface requires an explicit versioned schema, independent decoder/replay, hostile-input tests, bounds, and nonclaims.
- Never grant ambient filesystem, process, network, model, signing, payment, commit, publication, or registry authority merely because a declaration, receipt, test, or proof exists.
- Keep changes reviewable. Split implementation into commits that each preserve the existing test suite, and explain any known-answer update field by field.
- Do not weaken or delete a failing regression to obtain green CI. Fix the cause or narrow the admitted feature with a stable diagnostic.
Definition of done
The implementation, tests, documentation, status wording, diagnostics, and evidence agree on one exact scope. Targeted tests pass, the applicable repository quality gate passes, hostile cases fail with the documented stable reason, and no unrelated public surface or authority is widened.
Activity
feat(model-budget-policy): admit model calls, retries and failover ag…
wsdt commentedon Sep 12, 2026
Landed (
3afacaf4) — and this is not a duplicate of #113src/model_budget_policy/(classification,limits,provider_policy,ledger) plusdocs/MODEL-BUDGET-POLICY-V1.md. 38 selected, 38 passed incargo test --locked -p semaprax --lib model_budget_policy. Coordinator re-ranfmtandclippy --all-targets -D warningscentrally, both exit 0.What #113 actually covers, verified rather than assumed
Read
171a8751and the current files, and grepped for enforcement code:one monetary ceiling and one absolute deadline (nonrefundable by construction in
live_invocation::budget::CumulativeBudgetLedger), crash-safe resume by foldinga journal prefix rather than a second durable counter, five cancellation
checkpoints, and budget/deadline/cancellation kept as three distinct refusal
reasons.
The decisive finding:
approved_providersexists only as an identity-bindingfield — order matters because it feeds the derived
LiveInvocationId— andthere is no failover mechanism, no retry loop, no retry classification, and no
call/retry/provider ceiling anywhere in
live_invocation. "retry" and"failover" appear only in doc comments; zero enforcement code.
model_invoke.rs's own doc names theInvocationBudgetHookseam for "issues#113/#179". The two are complementary halves of it.
ModelBudgetLimits::max_callsAttemptOutcomeClass/retry_is_permittedProviderPolicy::admit_failoverModelBudgetLimitslimits::intersectordinal+AttemptKindper reservationNonrefundability, proven with numbers
a_failed_attempt_then_its_retry_both_spend_and_neither_is_ever_refundedasserts
cost_committed_microsmoves 0 → 30 → 30 (recording the failure doesnot refund) → 31 (the retry adds its own cost), and
aggregate_tokens_committed0 → 20 → 22, with distinct ordinals. Remainingcapacity decreases twice and is never restored.
a_self_reported_zero_cost_outcome_never_reopens_spent_capacitycloses theauthority hole: a recorded outcome claiming zero cost cannot reopen an exhausted
ceiling. Model data carries no authority.
Every budget driven to its exact boundary
one_millisecond_before_the_deadline_is_admitted/exactly_the_deadline_instant_is_refusedmax_calls: zero rejected at construction;the_call_exactly_at_max_calls_is_admitted_and_the_next_is_refusedmax_retries: exact-then-refused, gated byan_uncertain_prior_outcome_never_permits_a_retryanda_completed_response_never_permits_a_retry_eithermax_providers: exact-then-refused, plusfailover_out_of_the_deployments_exact_order_is_refused,failover_to_an_unauthorized_provider_is_refused_even_if_it_is_next_in_order, andfailover_is_refused_when_the_prior_outcome_was_uncertainaggregate_tokens_exactly_exhaust_at_the_ceiling_across_two_attempts_the_third_is_refuseda_negative_estimated_cost_is_refused_rather_than_treated_as_a_creditThe intersection tests carry the control that matters most:
max_retries_or_max_providers_at_zero_is_a_legal_restrictive_policy_not_a_rejection,paired with
an_intersection_collapsing_max_calls_to_zero_is_refused_as_zero_impossible.Without that pair, "zero is rejected" would be indistinguishable from "zero is
always rejected", and a legitimately restrictive policy would be unusable.
Uncertainty is respected throughout: an uncertain prior outcome permits
neither a retry nor a failover — the call may have happened, so treating it as a
free escape hatch would double-spend a real request.
Open
ModelPolicyLedgeris not wired into the live kernel loop orAgentDeployment— that downstream integration against the seam is #178–#181's,matching what
live_invocation's own docs describe. No live network, provider orkey; the existing public
InvocationClock/StepClockfixture is reused ratherthan a second clock seam invented.
Evidence class: local, offline fixtures. Not hosted.
style: apply cargo fmt to the model budget policy module
wsdt commentedon Sep 12, 2026
Audited against
mainatcc03e2e8. Not closing. The title says "Enforce", and nothing in the live system enforces anything yet.src/model_budget_policy/is merged intomainand is correct in isolation — butgrep -rl "model_budget_policy" srcreturns only its own module. Nothing insrc/live_invocation/kernel.rscallsModelPolicyLedger::reserve_attempt.docs/MODEL-BUDGET-POLICY-V1.mdsays so itself under "Out of scope": wiring the ledger into the kernel loop "is downstream work." So today no dispatch — not even a fixture dispatch — is actually bounded by this policy.limits::intersect,ledger::reserve_attempt— exact-boundary and limit-plus-one tests for every named dimension. No caller.provider_policy::admit_failover,classification::retry_is_permitted. The doc states the kernel has "no retry or failover primitive anywhere" today.uncertain_is_never_retry_permitted; a self-reported zero-cost outcome never reopens spent capacity. This is a proof about the taxonomy, not about live cancellation — that machinery belongs to #113.record_outcomeis evidence-only and never adjusts committed counters (a_failed_attempt_then_its_retry_both_spend_and_neither_is_ever_refunded)ModelBudgetLimits::validaterefuses a zero or negative policy before a ledger can be constructedEvidence class: local re-runnable, offline fixtures only. No live provider, credential, network call, or billing record exists anywhere in this code — that half is HUMAN_BLOCKED and cannot be produced by this run.
Remaining for closure: one caller. Until
live_invocation::kernelconsults the ledger on the attempt path, "enforce" is not a claim this issue can make.Clarify historical source journal gates against implemented v2 route
wsdt commentedon Sep 13, 2026
Landed live budget-hook integration on main at 852aa90.
LiveModelPolicyHookcomposes the existing model-policy ledger with the actual generic kernel work-budget seam. Explicit host quotes bind the exact request digest and declare context/output token and cost upper estimates. Stale/negative/overflowing quotes refuse before reservation. Intersected call/token/cost ceilings now stop actual dispatch; original-start latency becomes one checked absolute deadline, and cancellation/deadline are checked at existing kernel boundaries. Conservative model reservations remain charged if the later work gate refuses, and an altered effective work amount refuses rather than breaking quote identity.The actual two-turn kernel test dispatches once, refuses the second call, preserves all call/token/cost charges, and emits a compatible journal receipt. The streaming bridge test also uses this hook, so budget, compiled streaming decode and receipts are connected. Byte observations are never invented as actual token usage or actual cost.
Remaining: this hook retains in-process state only. Generic v1 journal bytes cannot restore token/cost reservations after a crash; source priced V4/V5 recovery remains separate. Generic turns are fresh attempts on an explicitly selected provider, not a durable retry/failover scheduler. Cross-route coverage and real usage/accounting remain open.
Local focused evidence: 56 selected library tests passed (one serial build), including the new integration paths, model-policy preservation tests, and receipt projection. Module-size, format and diff checks passed. No full quality suite, hosted CI, live provider or billing claim.
wsdt commentedon Sep 13, 2026
Pushed implementation batch 6b6d691 to main (starting at 02287bd).
The public RetryFailoverScheduler now drives actual SDK adapters through explicit factories, reserving each unique attempt before factory/start. It retries only declared safe classifications, keeps partial/uncertain/completed outcomes terminal, follows ordered authorized provider slots, preserves every reservation/result in a bounded in-process trail, and exposes host backoff without ambient clocks or waiting. The compiled route derives the existing exact adapter prompt from the retained checked schema and logical request, and rejects malformed final output. Cancellation/deadline checks cover factory/start/poll/settlement boundaries; both existing streaming bridges also recheck after a settlement poll returns.
Fixed standalone ledger provider checks for fresh/retry attempts and checked token addition. No reservation is refunded. Partial provider usage remains partial rather than becoming invented zero billing. Focused local evidence: 48 policy tests and 65 SDK tests passed, including actual compiled success, stale grammar/no-factory, malformed completion, safe retry, ordered failover and uncertain no-retry controls. No full quality or hosted gates were run, per maintainer speed instruction.
Still open: this is an in-process host scheduler, not durable generic journal recovery or a new authenticated DeploymentRoot policy route. Source/deployment/invocation cross-route enforcement and durable attempt restoration remain. No live-provider or billing claim. Logs: .agent-logs/0913-policy-corrected.log and .agent-logs/0913-provider-corrected.log in the private integration checkout.
wsdt commentedon Sep 13, 2026
Implemented and pushed
70e3ce9821facc9d6d37f7b5c425a572f95c2f9a(fromefa95897ca8be308ef054c0b083e8436dfab2dfb).The additive generic durable policy v2 now checkpoints intent before adapter construction and settlement afterward; unresolved intent never redispatches. Recovery validates canonical request/quote bytes, sequence, nonrefundable reservations, safe failure classes, streamed-byte contradictions, and exact capability requirements. Retained execution, source Proposal schema, task/budget, deployment provider/model selections, and seven source/deployment budget dimensions are checked before dispatch. The constructed adapter must declare the exact selected provider/model/capabilities before start. This declaration is not remote identity or billing proof.
Focused local evidence on the combined build: 6 journal tests, 4 durable kernel tests, 6 existing scheduler regressions, and 2 public retained-execution binding/dispatch tests passed (18 relevant tests). Commands used the existing library and
agent_runtime_v1harnesses with selectorslive_invocation::policy_journal::tests,live_invocation::policy_kernel::tests,model_budget_policy::retry::tests, andexecution_revision::durable_policy. Logs:.agent-logs/0913-combined-core-final.log,.agent-logs/0913-combined-agent-final.log. The latter also selected an unrelated #116 repair regression that failed; both #179 tests passed. No full/hosted/live-provider claim.Keeping this issue open. Remaining functional work includes durable cumulative request/response byte ceilings, complete malformed-output usage retention, and explicit treatment of aggregate-token authority where the retained source profile has no matching token dimension. A post-failed-poll cancellation check and an additive byte-accounted profile are in progress; v2 wire bytes remain frozen.
wsdt commentedon Sep 13, 2026
Accepted bbf9a95 → c02b49e on main (implementation a644d37; preserved concurrent README-only b33a8af).
Added an executable durable V3 byte-accounting route over the existing retry scheduler. It derives exact serialized SDK envelope sizes through the scheduler projector, intersects retained source/deployment byte ceilings with narrower invocation limits, and persists nonrefundable per-attempt and cumulative reservations before factory creation. The outer checkpoint preserves frozen canonical V2 bytes; replay reconstructs accounting, rejects altered/noncanonical bytes, and never redispatches an unresolved intent. Failed or settled output above its reservation refuses the outcome and leaves the intent uncertain. Atomic append controls prevent partially updated byte ledgers.
Post-poll cancellation is checked even when the poll returns an otherwise safe zero-byte failure. The regression proves one charged factory attempt and no retry/fallback.
Local focused evidence: 4 V3 journal tests, 4 executable V3 route tests, 3 byte-ledger/binding tests, 1 post-poll cancellation test, and 2 retained execution/schema/model binding tests passed. Logs .agent-logs/0913-v3-{journal,route,bytes,cancel,retained}.log. An independent review checked envelope equality, retained-key compatibility, recovery and ACK order. No full/hosted/provider-billing claim.
#179 remains open: this does not establish remote provider cancellation or actual reported-usage completeness after malformed schema settlement; broader end-to-end budget evidence remains distinct from these local durable guarantees.
wsdt commentedon Sep 18, 2026
Already implemented — verified against code and executing tests, closing
This issue's audit baseline is
ae25c6a4(2026-09-11) and it warns that latercommits may have superseded individual details. They have. A full re-audit of
the eight implementation steps and all seven required-evidence items against
current
mainfinds every one satisfied by code with a passing executable test.No code was changed to close this.
Implementation sequence
ModelBudget v1/ModelRetryPolicy v1, closed classification, usage-stage separationAttemptOutcomeClass(src/model_budget_policy/classification.rs:30-61) is the closed 5-way set;AttemptRequest/AttemptReservation/AttemptUsage(ledger.rs:64-148) separate requested/reserved/attempted/accepted;TokenAccountingSource(src/provider_adapter_sdk/capability.rs:53-63) +BillingReconciler::reconcile_provider_reported(model_call_receipt/reconciliation.rs:333-432) add provider-reported vs reconciledintersect(limits.rs:157-167) elementwise-mins all three thenvalidate()?.EffectiveModelBudget(limits.rs:143) has no bare public constructor and I confirmed zero in-crate bare constructions outsidelimits.rs, so an unvalidated budget cannot reach a handlerModelPolicyLedger::reserve_attempt(ledger.rs:308-434) commits nonrefundably before provider contact;RetryAttemptJournal::intentatretry.rs:616precedesadapter.start()atretry.rs:852retry_is_permitted(classification.rs:86-93) admits onlyNotDispatched,RejectedBeforeProcessing,ProviderReportedRetryable;retry.rs:1009-1018additionally intersects that with the adapter's own declaredcapabilities().retryable_failure_classesRetryBackoff(retry.rs:88-90) is an injected pure callback; the scheduler never reads a clock or sleepsProviderPolicy::admit_failover(provider_policy.rs:108-129) enforces exact order and confidentiality authorization;DurablePolicyBinding::bind(durable.rs:73-286) binds toExecutionRevision/DeploymentRoot;AttemptReservation.ordinalstrictly increasing (ledger.rs:84-93)retry.rs::drive(:783,:836,:865,:894), always mapping toUncertain/NotDispatched, never a false "did not happen"AttemptRefusal(ledger.rs:98-137) names the dimension and the numbers (CallsExhausted,AggregateTokensExhausted{requested,remaining},CostExhausted{...}) and propagates viaSchedulerRefusal::PolicyRequired tests and evidence
ledger.rscarries the exhaustive_exactly_at_the_ceiling..._next_is_refusedfamily across calls, retries, providers, context tokens, output tokens, aggregate tokens, cost and deadline.negotiate()gate (retry.rs:828-835),limits.rsrejection tests,public_compiled_route_refuses_stale_grammar_before_factory,a_pre_cancelled_attempt_is_refused_and_commits_nothing.an_uncertain_prior_outcome_never_permits_a_retry,a_completed_response_never_permits_a_retry_either,an_uncertain_post_start_failure_never_constructs_a_fallback_adapter.an_unauthorized_next_in_order_provider_is_refused_and_never_silently_skipped,failover_to_an_unauthorized_provider_is_refused_even_if_it_is_next_in_order,failover_out_of_the_deployments_exact_order_is_refused.retry/tests/cancellation_tests.rs:34 cancellation_after_poll_prevents_a_zero_byte_safe_failure_retryasserts the factory ran exactly once andrun.attempts.len() == 1with the comment "the pre-dispatch reservation remains charged". I read this test body directly to confirm it proves retention rather than merely naming it.retry.rs:1448 malformed_completed_response_is_not_reported_as_public_successclassifiesCompletedWithResponse(charged), not a silent retry.src/live_invocation/policy_kernel/tests/v3_tests.rs:47 v3_settles_then_terminal_replay_constructs_zero_adapters, assertingreplay_calls == 0at:111.Verification run
Nonclaims and residual scope
ScriptedAdapter,StepClock, fixture factories. Nothing in this module has contacted a live provider, by design: the capability invariant forbids ambient network authority, and credentials belong to the host via the adapter seam. This is not hosted or production evidence and is not described as such. A live-provider run belongs to the adapter integrations ([P1] Create a provider adapter SDK and deterministic model-provider conformance suite #181), not here.DurablePolicyBindingRefusalhas 14 variants and roughly 5 have direct per-variant coverage. The uncovered ones (EmptyProviderPolicy,ProviderOrderMismatch,ProviderIdTooLong,DeadlineOverflow,SchemaMismatch,ProviderNotDeploymentSelected,ContextLimitExceedsDeployment,AdapterRequestBytesExceeded) are structurally simple guards. The issue's required-evidence list does not demand per-variant exhaustiveness, so this is noted rather than treated as blocking.EffectiveModelBudget's inner field ispub(crate). No in-crate code bypassesintersecttoday (verified by grep), but the type does not structurally prevent a future in-crate bypass. Worth a follow-up only if that becomes a real risk.Closing as already satisfied.