feat: add scripted model test utilities - #4362

The corpus now carries 1.18 fixtures emitted by the v0.23.0 tag itself, both
the minimal payload and a pending write holding the compaction metadata that
release published, so the reader branch for the released label is covered by
historical-writer output instead of a relabeled canonical payload. The held
pending write moves to 1.19 fixtures emitted by this branch's own bump commit.

Only these entries were regenerated. tests/fixtures/run_state/generate_corpus.py
cannot rebuild the whole corpus in this checkout: the pending-approval scenario
pins writer 92aa1b9 (2026-08-05) while its code imports agents.testing, which
landed in openai#4362 on 2026-08-13, so that scenario fails before it can emit.

* fix(sessions): persist deferred interrupted-turn items when the approval resume continues the run

With output guardrails and a non-default tool_use_behavior,
_should_defer_interrupted_session_items defers the interrupted turn's session
items at interruption time. When the approval resume resolves into
next_step_run_again (or a handoff), the resume-side write only carried the
resolved turn's new items - the tool output - and no later write recovered the
deferred function_call. The Session ended up with a function_call_output whose
call was never persisted, and the Responses API rejects every later run over
that Session with 'No tool call found for function call output'.

Persist the deferred prefix (the current response's session items, located via
the resumed response boundary) ahead of the resolved turn's items once the
resume commits to continuing the run, in both the streamed and non-streamed
paths. The final-output path is untouched: its persistence already reconstructs
the full current response. A resume that interrupts again keeps deferring.

* refactor: share the deferred-prefix selection between both resume paths

Codex review: the selection lived twice, once per resume path, and AGENTS.md
wants runtime logic under run_internal. It now lives next to the gate that
governs it (_deferred_interrupted_session_prefix in blocked_output.py) and both
paths call it.

* fix: derive the park-time deferral decision from checkpoint state

Codex review: re-evaluating _should_defer_interrupted_session_items against the
live configuration at resume time loses the deferred function_call again when
the caller resumes with tool_use_behavior='run_llm_again' (reproduced before
changing anything). A non-deferred interruption write bumps
_current_turn_persisted_item_count, so persisted_count == 0 identifies the
deferred park on its own — the helper now keys on checkpoint state only, which
also keeps the prefix empty (no double write) when the interruption-time write
actually ran. Two tests: the behavior-change resume, and the non-deferred park
not being written twice (mutation-checked: dropping the persisted-count guard
turns it red).

* fix: carry the deferred prefix through a streamed re-interruption

Codex review: a resume can interrupt again (partial approval of a
multi-approval response). If the gate no longer defers, that re-interruption
write is the deferred prefix's last chance — it bumps the persisted count, so
writing only the approved tool's output there orphaned BOTH parked calls for
every later resume (reproduced: 2 orphans before this change). The streamed
re-interruption branch now prepends the prefix exactly as the non-streaming
path already did; a gate that still defers keeps deferring, and the
still-deferring variant recovers everything at final output (verified).
Regression test proven red against the previous commit.

* fix: confirm the deferred prefix against the Session and carry it to every resume exit

Three defects in the previous commits, each reproduced before changing anything:

1. A resume whose approved tool IS terminal ends in final output, and
   _final_turn_items_for_persistence only rebuilds the current response when the
   agent has output guardrails — a resume may legitimately run without the ones
   the park had, and the parked function_call was dropped again (both runners).
   The prefix now rides that exit too.

2. persisted_count can legitimately lie: the resumed-safety validator resets it
   to zero for a DETACHED resume, and that reset outlives the run, so a later
   resume reconnecting the original Session rewrote items it already held
   (duplicate function_calls, measured). The prefix is now CONFIRMED against the
   Session's own tail using the existing fingerprint helpers, so the write is
   idempotent by construction and a detached resume degrades to writing nothing.

3. An empty resolved turn (a handoff input_filter can drop every item) must not
   strand the prefix on its own: a call written without its output poisons the
   Session exactly as the orphaned output does. It keeps deferring instead, and
   both runners now agree on that.

The helper moves to session_persistence, where the Session read and the
fingerprint helpers already live, and becomes async. Five new tests, the three
new ones proven red against the previous commit.

* fix: suppress only what the Session provably holds, by collision-free identity

Codex review: filtering the prefix against an unordered set of content fingerprints
drops an item that merely LOOKS like one already there. An assistant preamble
repeats verbatim across turns, so a tail holding an identical preamble from an
EARLIER turn made the current one vanish while its calls were still appended —
a legitimate occurrence lost from history (reproduced before changing anything).

Matching the whole prefix as an ordered block was the obvious answer and is wrong
too: a partially written response (calls persisted by an earlier attempt, output
not yet) then matches nothing and duplicates the calls. Measured, both ways.

So suppression is now keyed on identity that cannot collide — (type, call_id),
unique per turn — and anything without one is kept unconditionally. A partially
written response contributes exactly its missing half; nothing is ever dropped for
looking familiar. Regression test proven red against the previous commit.

* fix: recognize hosted MCP approval identities when reconciling the prefix

Codex review: not every item family names its id 'call_id'. A hosted MCP approval
request identifies itself with 'id' and its response points back with
'approval_request_id', so _identity_key returned None for both (measured) and a
partially written response would append requests the Session already holds —
duplicate request ids corrupt the history the next model call reads.

The request identity is read through get_hosted_mcp_approval_request_identity,
the repository's canonical helper, rather than a local rule. Request and response
keep DISTINCT identities (same id, different type), so persisting one never
suppresses the other. Items with no collision-free id still return None and are
therefore never suppressed.

Regression test proven red against the previous commit.

* test: pin that an emptied resolved turn corrupts nothing in either runner

Review follow-up. The empty-turn shape (a handoff input_filter drops every item
of the resolved turn) must not write the deferred prefix on its own: a call with
no output poisons the Session exactly as the orphaned output does. Both runners
must also agree item for item, because a divergence here is how a dangling-call
regression first shows up.

Proven red against 9ba9fef, where the streamed path wrote both calls dangling
(call_PARKED, call_HANDOFF) with no outputs.

Also measured for the review discussion, not encoded in the test because they
are properties of dead code: the RunAgain clear at run.py never executes in this
shape (the emptied turn resolves into a handoff), RunAgain with empty session
items is not constructible (approve and reject both yield an output item), and
deleting the clear outright changes nothing across all reproducers and the full
suite. The batch loss itself is real in BOTH runners and is the same root as the
call-id collision finding: only the serialized checkpoint can carry it (#4827).

* fix(sessions): declare the withheld interrupted write as a held pending Session write

The output-guardrail persistence gate withholds the interrupted turn's session
write at park time. The previous approach reconstructed that batch on resume by
reconciling the checkpoint against the Session's history, which broke context
wrapper propagation, legacy sessions, and detached reconnects, and could not
prove what the park had withheld.

The park now registers the withheld batch on the existing
RunState._pending_session_write slot with a held marker. Registering is not
writing: the Session is only touched at a gate-legal exit of a later resume,
where the batch lands ahead of the resolved turn's items in one ordered append
and inherits the digest-based crash recovery. A run-again checkpoint settles at
entry, a detached exit folds the resolved items into the standing batch, an
emptied resolved turn discards it, and the blocked-output redaction never sees
it raw. The history-reconciliation machinery is deleted.

* test: cover the held pending write across both runners and its serialized contract

The acceptance battery drives park, approve, reject, re-park, detached carry,
tripwire, guardrail crash, emptied turn, legacy and context-aware sessions, and
a failed settle recovered on the next resume, each through both runners and a
serialized checkpoint. The resume-path suite pins the held marker's validation
and that a checkpoint without the marker keeps its released eager-settle
meaning.

* refactor(sessions): drop the dead no-state park bridge and tighten defer signature

The non-streamed runner always builds a RunState for a fresh run, so the
interruption result reads the held record from the state itself; the extra
carrier parameter could never be exercised.

* test: pin that a settled or discarded batch never lingers on the live state

* style: apply ruff formatting to the touched files

* fix(sessions): narrow the fresh park registration for the type checker

* fix(sessions): validate the held resume's Session and settle only paired held calls

A held checkpoint resumed against a different Session must fail at boot like an
ordinary pending write, before the approved tool can execute and the batch can
settle into the wrong conversation. And a handoff input_filter may drop a subset
of the resolved outputs, so the settling batch being non-empty does not make it
safe: a held call settles only when its output survived, which honors the
filter's decision symmetrically in both directions.

* style: satisfy line length and the settling batch parameter type

* fix(sessions): carry the held batch through detached parks, pending approvals, and completions

Four holes the adversarial pass over the final diff surfaced, each reproduced in
both runners before fixing. A fresh park during a detached resume now folds the
new call into the standing held batch instead of losing it. The pairing guard
exempts calls whose approvals are still open on the current step: their outputs
are missing because they have not run yet, not because a filter removed them, so
they settle like a non-deferred park writes a call before its output. The entry
settle applies the same pairing contract against the batch alone. And a detached
completion discards the batch at the fresh final exit too, so a completed run's
checkpoint stays loadable and the runners agree. The streamed resume test double
now forwards the settling batch.

* fix(sessions): keep run input out of the held batch, register final settles, and restore Conversations sanitization at entry

Three findings from the third automated review round. The sandbox runtime defers
the pre-turn input save, so a deferred park used to fold the Session's only copy
of the accepted input into the held batch, where a tripwire discard would take
it along: the deferred arm now persists any unsaved input exactly as the
non-deferred arm does, and the batch carries only the withheld response. The
final-output settle now registers the claimed batch before appending, so a crash
inside that append fails closed with the batch recorded instead of silently
losing it. And the attached entry settle re-applies the Conversations-specific
sanitization a detached extension could not, restoring the backend invariant
before the direct append.

* fix(sessions): dedupe the rebuilt final response against the batch and pair every approval family

Two more findings from the same review round. With output guardrails the final
sweep rebuilds the whole current response, held batch included, and the item
deduplication cannot key the batch's unkeyed companions, so an assistant
preamble landed twice: when the final items already carry every held request the
batch is redundant and is dropped, in both runners. And the pairing guard now
speaks every supported approval identity, hosted MCP requests and responses
included, instead of recognizing only the function-call pair.

* fix(sessions): settle held batches through the canonical pairing and counting rules

Four more findings from the same review round. The held pairing guard now
delegates to the canonical drop_orphan_function_calls, so every tool-call family
in _TOOL_CALL_TO_OUTPUT_TYPE pairs (shell and apply-patch included) and a
reasoning item riding before a dropped call is pruned with it, as the Responses
API requires. A Conversations-backed registration forces the reasoning-id policy
to None like the normal save, so a server-identified reasoning item stays
persistable. And settled held items count toward the turn's persisted count, so a
later gate-enabled resume fails fast on the persisted-items refusal instead of
re-appending the stored calls.

* fix(sessions): keep crash recovery armed when the rebuilt final items cover the held batch

The guardrail rebuild deduplicates the held batch out of the append to avoid
doubling its unkeyed companions, but the append still lands the approved call
and output, so the recovery registration must stay armed. Arming now keys off
whether a held batch was claimed at all, captured before the dedup empties the
payload, in both the resumed-turn helper and the zero-count final save; a crash
inside the append leaves the batch recorded to reconcile on retry instead of
silently losing it.

* fix(sessions): give the held pending write its own schema version, settle it through the canonical path, and close the terminal and commit-boundary gaps

Recovered work addressing the four blocking review points:

- The held variant adds keys the released 1.17 reader rejects by exact key set,
  so it now has its own schema version. 1.17 keeps its four-key form and its
  original summary; held and response_id are gated to 1.18, with corpus
  fixtures, sources, README and the version-boundary test updated.
- The entry settle no longer appends behind the canonical persistence path: it
  goes through save_result_to_session like every other settle, inheriting the
  Conversations sanitization, the ordered dedup, the pending-write registration
  and the compaction bookkeeping for the response the batch belongs to, which
  the park now records.
- A max-turn handler ends the run, so both runners discard a held batch there.
- The resumed turn's output committer folds a committed tool output into the
  held batch, so a post-output callback that raises cannot leave a retry that
  skips the completed invocation and drops the executed call and its result.

* fix(sessions): defer compaction when the settling batch carries the tool output

The consolidated settle hands the held batch to the canonical path through the
original_input slot, but the deferral decision only inspected new_items, so a
batch containing the approved tool's output reported no local tool output and
compacted the very response whose output had just landed. The decision now asks
whether the append persists a local tool output at all, whichever slot carried
it, and the batch records the store setting of the turn it was withheld in so
the deferral resolves the same compaction mode the ordinary path would.

* fix(sessions): make the 1.18 corpus entries reproducible and scope the held-only keys

The corpus entries claimed a 1.18 writer for a commit that emits 1.17 and has no
response_id, and the generator had no 1.18 scenario, so regenerating the corpus
would have dropped them. Both fixtures are now what the recorded 1.17 writer
emits with only the schema label changed, the generator carries the matching
scenarios, and the README says the same thing. The reader also refuses
response_id and store on an ordinary pending write, where they describe nothing,
and the schema rationale no longer claims 1.17 shipped in a release: its readers
are on main, which is reason enough not to rewrite what they already emit.

* test: pin that the held-only keys are refused on an ordinary pending write

* fix(sessions): settle the pairs an emptied resolved turn leaves behind

A handoff input_filter can drop every resolved item, but by then the approved
tool has run and its output was folded into the held batch. Emptiness of the
turn was the wrong predicate: pairing is. The executed call and output now
settle through the canonical path and only the unpaired requests drop, so the
Session keeps the only record that the tool ran and a later run does not
re-issue its side effect.

Three defects the same review surfaced go with it:

- The compaction deferral read the whole ``original_input`` slot, which carries
  the caller's own input on an ordinary save. Only a settling batch reads it now.
- The settled count added the batch's raw length, overcounting whatever the
  dedup dropped; the append reports what it actually wrote, and that count
  slices a later save of the same turn.
- The max-turns discard ran before ``validate_handler_final_output``, so a
  wrongly typed handler output lost the batch that the streamed runner keeps.

Each is pinned by a test proven red against the previous behaviour.

* fix(sessions): dispose of the held batch when the run ends, not when its last step is chosen

Choosing a terminal step is not the same as ending the run. Validation, the
final-output hooks, the output guardrails and the final save all run after that
choice, any of them can raise, and a run that raises may still be retried or
reattached with the approved tool's call and output reachable only through the
held batch. Consuming the batch at the choice threw it away on every one of
those failures.

Four sites carried that ordering, and they are the whole class: the max-turn
handler finalization, the detached final output in both runners, and the
detached final output on the resumed streamed loop. Each now disposes of the
batch once finalization has completed, with a tripwire handled separately as
the decided blocked outcome it is. The five remaining disposal sites are
deliberate ones that follow an outcome already decided, and they are unchanged.

Also: a re-interruption no longer overwrites the storage setting the parked
response was produced under. Presence of the key decides, not its truthiness, so
an ordinary ``store=None`` park keeps its own setting and the settle resolves
that response's compaction mode from the right turn. ``response_id`` follows the
same rule for the same reason.

Tests pin the failure of each finalization stage in both runners, and the park
storage settings across None, False and True.

* fix(sessions): count and classify the settling batch on the compaction path

Two defects in how a settling held batch meets a compaction-aware session:

- The compaction-deferral branch returned the run-item count alone, and it is
  the branch every held settle with outputs takes on such a backend, so exactly
  the sessions that defer were the ones whose settled turns undercounted. The
  count gates the resumed-safety refusal and slices later saves of the same
  turn, so it must equal what the append wrote. The branch now returns the
  combined count.

- The local-continuation classification knew the mapped tool outputs but not
  the hosted MCP approval response, which is the locally produced half of its
  approval pair and must stay associated with the response chain that carried
  the request. Compacting that response before the model consumes the approval
  drops it in previous_response_id mode. The constant is now
  _LOCAL_CONTINUATION_OUTPUT_TYPES and covers both carriers: the settled dict
  and the MCPApprovalResponseItem the non-deferred resume commits, because
  classifying one and not the other would defer or compact the same response
  depending on which path persisted it.

The four-stage scenario behind the count (partial approval, held settlement
with a lapsed gate, gate re-enable, remaining approval) is pinned end to end:
it must end in the documented fail-fast refusal with nothing duplicated. Each
fix is also pinned at the unit boundary and proven red by mutation.

* fix(sessions): the held record owns the conversion policy of its items

A detached re-park cannot see the Session backend, so it folded new items under
the resuming run's own reasoning-id policy. For a Conversations-origin batch
that strips the server id at the one point where nothing can restore it, and
the reattach then drops the reasoning item as unpersistable.

The park now records the conversion policy it actually used (None for a
Conversations backend, the run's policy otherwise) on the held record, and a
fold converts under the record's policy instead of the caller's. The key is
gated and validated with the other held keys under the unreleased 1.18 schema,
refused on ordinary pending writes and on unknown values; an absent key falls
back to the caller's policy.

Pinned at the fold and at the park, plus the validator rejection, each guard
proven red by mutation.

* fix(sessions): the final sweep's direct settle speaks the settle dialect

A reattached detached carry can reach the final exit with a zero persisted
count, so the batch settles through the final sweep's direct save. That call
armed the recovery registration but not the settle marking, so the compaction
deferral could not see the batch's outputs when the final turn carried none of
its own, and the returned count excluded what the settle wrote. One flag closes
both, pinned at the helper boundary and proven red by mutation.

Also repairs the held-record docstring, whose conversion-policy paragraph had
split a sentence in two.

* fix(sessions): held settlement derives pairing evidence from the filtered session view

``HandoffInputData.new_items`` is the session-history axis by contract, and
``input_items`` exists precisely so a filter can shape model input while
preserving history. The held settle defeated that: an output the commit
boundary folded into the batch settled on the batch's own evidence, so a filter
that removed a complete pair from ``new_items`` — the library's own
``remove_all_tools`` included — found it persisted anyway.

The commit boundary now records the call ids it folds in a deliberately
transient, per-turn set on the run state. A settling or folding exit whose
``run_items`` are the resolved session view drops the batch's copy of those
outputs: kept ones arrive through the view itself, removed ones must not land.
The set resets at each turn boundary, because the filter's authority covers one
turn, and it does not serialize, because an output folded by an earlier process
is carried prior-turn history that a later turn's filter is not entitled to
remove, exactly as the eager path cannot unpersist earlier turns. No schema
change.

The regression pair Sylvester asked for is pinned in both runners, plus the
library-filter secret case, the detached filtered handoff riding a checkpoint,
and the carried-pair boundary. Two prior tests asserted the settled-from-batch
behaviour and now pin the corrected contract.

* fix(sessions): fold ownership rides the held record so the filter contract survives a serialized retry

A post-output callback crash leaves the folded output on the checkpoint, and
the supported retry path serializes and reloads that state. The fold's
ownership lived in process memory, so the reloaded retry could not tell the
crashed turn's own outputs from carried history: a handoff filter that removed
the pair found the batch settling it anyway, the library's own
``remove_all_tools`` included.

The commit boundary now records the call ids it folds on the held record
itself (``folded_tool_outputs``, gated and validated with the other held keys
under the unreleased 1.18 schema), together with the turn that owns them, and
the in-memory set is gone. A settle or fold on an exit where a
``Handoff.input_filter`` actually ran (``SingleStepResult`` now says so) drops
the batch's copies of the owned outputs: kept ones arrive through the resolved
view in the same save, removed ones must not land, and the pairing prune takes
their calls with them. Exits without a filter are untouched, so a partial
approval's second resume still settles the earlier approval's pair from the
batch, and the marker expires with its turn, so carried prior-turn history
still settles at the reattaching entry.

Pinned by the serialized-retry regression in both runners alongside the
existing filter-contract pair, the partial-approval flow, and the carried-pair
boundary; the marker's validator shape has its own rejection case; each guard
proven red by mutation.

* fix(sessions): the filter's authority covers the batch's unkeyed companions too

The held batch carries the parked response's unkeyed companions (an assistant
preamble, an id-less reasoning item), and a handoff filter that removed them
from the session view found them persisted from the batch anyway: the gate
covered the owned outputs and nothing else.

On a filtered current-turn claim or detached fold, an unkeyed batch item now
survives only if the resolved view kept it, matched by the same fingerprint the
dedup uses. Calls and request kinds stay with the pairing rule, owned outputs
keep their unconditional drop, and carried prior-turn history remains outside
the filter's reach. Pinned by the filtered-preamble regression in both runners,
each branch proven red by mutation.

* fix(sessions): relabel checkpoints as 1.17 the way the released reader reads them

Main bumped the schema to 1.18 for agent-scoped approvals and MCP recipient
bindings while this branch was in review, so the held pending write now shares
the unreleased 1.18 label instead of taking a number of its own, the same way
1.17 absorbed the pending Session write before it. The two tests that relabel a
current checkpoint as 1.17 must therefore also strip the 1.18-only context and
response fields, exactly as main's own relabeling tests do, or the reader
rejects the payload before the pending-write validator ever runs.

* fix(sessions): retain accepted held history through finalization failures

* fix(sessions): centralize resumed persistence and preserve filtered history

* fix(sessions): preserve response storage policy across approvals

* fix(sessions): synchronize turn before detached approval park

* fix(sessions): settle held history on empty streamed retry

* test(sessions): pin the held write to the label that introduced it

v0.23.0 released schema 1.18 with the compaction metadata on the pending write
and a reader that validates it by exact key set. A held record under that label
is therefore rejected by the released reader, so these tests pin the contract
seratch asked for: compaction metadata keeps loading under 1.18, a held
checkpoint emits the version that introduced the held keys, and relabeling it as
1.18 or 1.17 is refused. The emit and relabel cases are red on the current head.

* fix(sessions): move the held pending write to schema 1.19 behind a split validator

v0.23.0 shipped RunState schema 1.18 with the compaction metadata on the
pending write, and its reader validates that object by exact key set. The held
keys arrived after that release, so writing them under the 1.18 label emits
checkpoints the released reader rejects and misidentifies the durable format.

The held pending write now introduces 1.19. The validator gates each key to the
version that introduced it: the four base keys from 1.17, the compaction
metadata from 1.18, and held, reasoning_item_id_policy and current_response
from 1.19, so a released 1.18 payload keeps loading while a held record is
refused under any older label. The 1.18 summary is restored to the text v0.23.0
published and 1.19 carries the held write. Version assertions compare against
CURRENT_SCHEMA_VERSION or the version tuple instead of a 1.18 literal, and the
corpus generator records v0.23.0 as the historical 1.18 writer and this commit
as the 1.19 writer; the regenerated fixtures follow in the next commit.

* test(sessions): record v0.23.0 as the historical writer for schema 1.18

The corpus now carries 1.18 fixtures emitted by the v0.23.0 tag itself, both
the minimal payload and a pending write holding the compaction metadata that
release published, so the reader branch for the released label is covered by
historical-writer output instead of a relabeled canonical payload. The held
pending write moves to 1.19 fixtures emitted by this branch's own bump commit.

Only these entries were regenerated. tests/fixtures/run_state/generate_corpus.py
cannot rebuild the whole corpus in this checkout: the pending-approval scenario
pins writer 92aa1b9 (2026-08-05) while its code imports agents.testing, which
landed in #4362 on 2026-08-13, so that scenario fails before it can emit.

* test(sessions): downgrade only what the older label cannot represent

The relabel helper rewrote agent-scoped approvals and MCP recipient bindings for
every target label, but schema 1.18 supports both. Relabeling a current payload
as 1.18 therefore handed the reader a 1.17-shaped payload wearing a newer label,
and the test asserting that the released 1.18 format still loads would have
stayed green through a regression in that reader. The helper now downgrades only
below 1.18, and the held-gate test relabels the batch untouched so the held keys
are the only possible reason for the refusal. Both are proven red against a
reader whose approval gate moves off 1.18.

The generator and corpus README now record that the 1.19 writer SHA lives on
this branch and must be re-pinned to the squashed merge commit, the same way
821afdc stopped resolving for the entry this one replaces.

* fix(sessions): preserve model order after callback recovery

---------

Co-authored-by: Justin Beckwith <[email protected]>