Skip to content

Commit cd437f0

Browse files
fix(sessions): recover fresh streamed handoffs after session append failures (#4835)
* fix(sessions): recover fresh streamed handoffs after session append failures start_streaming()'s generic-loop NextStepHandoff branch awaited the fallible session append (_save_stream_items_without_count) before updating current_agent, run_state._current_agent, streamed_result.current_agent, and run_state._current_step. If that append raised (e.g. a transient session backend error), the run failed with those fields still pointing at the pre-handoff agent, even though the handoff had already fully executed. Resuming from result.to_state() after such a failure then re-invoked the wrong agent with input that already contained its own handoff call/output. This is the same defect PR #4725 fixed in the sibling is_resumed_state branch (used when resuming an interrupted run) by moving the state updates ahead of the fallible save. This applies the same reordering to the generic branch, which every fresh streamed run's handoffs go through, not just resumed ones. * address Codex review: checkpoint the handoff batch and reorder the agent-update event Two issues from the automated Codex review on this PR, both real: 1. _save_stream_items_without_count() never registered a pending_session_write checkpoint for the handoff batch, unlike the sibling is_resumed_state branch's _save_resumed_stream_items(). A failed append followed by a successful resume invoked the correct (delegate) agent but permanently dropped the handoff's function_call/function_call_output pair from session history, since nothing recorded the batch for the existing resume_pending_session_write() recovery path to replay. Fixed by threading resumed_write_state through _save_stream_items into save_result_to_session, gated on the handoff branch already having set _current_step to NextStepRunAgain. 2. AgentUpdatedStreamEvent was still queued after the fallible session append, so a live stream_events() consumer would see handoff items followed directly by an error with no semantic agent-transition event, even though the result and resumed run both correctly identify the new agent. Moved the event queue call to sit with the other state-transition updates, before the append. Both fixes are scoped to only the generic-loop branch this PR already touches; the already-merged is_resumed_state branch (#4725) has the same pre-existing event-ordering gap but is out of scope here. Extended test_fresh_streamed_handoff_preserves_agent_after_session_append_failure with a session-history assertion for issue 1, and added test_fresh_streamed_handoff_publishes_agent_update_before_session_append_failure for issue 2 (using a new session double that yields before failing, since a purely synchronous raise never gives stream_events() a scheduling boundary to prove event delivery either way). * address 2nd Codex review round: guardrail race, event drain, deferred compaction Three more issues from the automated Codex review on commit 75fc64b, all verified with live reproduction before being fixed: 1. The handoff transition committed current_agent/run_state before a still-in-flight parallel input guardrail had resolved. A non-tripwire exception from that guardrail then left the resumable state pointing at the delegate agent, even though the starting agent's input guardrails never definitively cleared. Fixed by explicitly awaiting input_guardrail_tripwire_triggered_for_stream() as the first statement in the handoff branch, before any state mutation. 2. Queuing AgentUpdatedStreamEvent before the fallible session append doesn't guarantee delivery: stream_events() checks a stored exception before draining the queue, so a real (non-instant) consumer can lose an already-queued event to a task that raised without ever being marked for draining. Fixed by marking the session-persistence exception via _mark_error_to_drain_stream_events() before re-raising, the same pattern already used for model-behavior errors. 3. The pending_session_write checkpoint recovers the raw item append but never carried enough information (response_id, store, whether the batch had local tool outputs) for a later, separate resume to replay the same post-write Responses compaction decision save_result_to_session would have applied inline. Extracted the compaction decision into a shared _apply_post_write_compaction() helper, extended the checkpoint schema with those fields (optional, so an old-shaped serialized RunState still round-trips), and call the helper from resume_pending_session_write() once a checkpoint settles -- whether inline or on a separate resume -- instead of duplicating the call at both sites. Added 3 new regression tests to tests/test_run_impl_resume_paths.py (72 total in the file, up from 69), each confirmed to fail against the pre-fix code and pass after. Full verification stack clean: make format/ lint/typecheck, and the full suite (9372 passed, 33 skipped, 0 failed). * address 3rd Codex review round: retain checkpoint until compaction settles One more issue from the automated Codex review on commit 92891a6, verified with live reproduction before being fixed: resume_pending_session_write() cleared run_state._pending_session_write before calling the newly-added _apply_post_write_compaction(), so if that call raised or was cancelled, the checkpoint was already gone. A later retry would then have nothing to redo the compaction step with, silently and permanently losing the requested deferred/forced Responses compaction even though the append itself had already succeeded. Fixed by moving the compaction call inside the try block, before clearing the checkpoint. The append reconciliation above already makes a retry safe against duplicate appends (it detects an already-committed batch via digest matching and skips re-appending), so this only changes when the checkpoint is released, not the retry logic itself. Added test_fresh_streamed_handoff_retains_checkpoint_when_post_write_compaction_fails to tests/test_run_impl_resume_paths.py (73 total, up from 72), confirmed to fail against the pre-fix code (checkpoint cleared despite the compaction failure) and pass after. Full verification stack clean: make format/lint/typecheck, and the full suite (9373 passed, 33 skipped, 0 failed). * address human review: clear the deferred-compaction marker only after success One more issue, this time from a human reviewer (sylvesterkaczmarek) on commit 06e9e40, verified with live reproduction before being fixed: OpenAIResponsesCompactionSession.run_compaction() cleared self._deferred_response_id before calling the fallible client.responses.compact() API. If that call raised, the deferred marker was already gone. The checkpoint-recovery code added in the last two commits recomputes force=True purely from whether this marker is still set, so on retry it silently recomputed force=False and could skip compaction that was still owed -- even though the round-3 fix already let the checkpoint itself survive for a retry. Fixed by moving the clear to after compaction actually settles (after the API call and the underlying session replacement both succeed), not before attempting them. The digest-based retry-safety already added for the append doesn't need any changes; this only moves when one session-internal flag gets cleared. Added test_run_compaction_retains_deferred_marker_when_api_call_fails to tests/memory/test_openai_responses_compaction_session.py, confirmed to fail against the pre-fix code (assert None == 'resp-handoff') and pass after. Full verification stack clean: make format/lint/typecheck, and the full suite (9374 passed, 33 skipped, 0 failed). * fix(streaming): stop the handoff transition on a real input-guardrail tripwire The generic-loop handoff branch already awaited an in-flight parallel input guardrail before committing the transition, but discarded its boolean result. A guardrail that settled normally with tripwire_triggered=True (no exception) therefore still published the delegate and a resumable NextStepRunAgain -- the InputGuardrailTripwireTriggered exception only fired later, from the pre-existing end-of-run safety net, by which point the transition had already leaked to stream consumers and to current_agent. Capture the boolean and raise immediately when it's true, before any part of the transition (current_agent, run_state, published events, session save) commits -- matching the exception-raising branch this same block already had for a guardrail that raises outright. * fix(streaming): trim speculative handoff items on an input tripwire Not raising the handoff transition on a tripwire (1a24069) doesn't undo it: the turn's model response, generated items, and session items are accumulated into streamed_result (and run_state, when resuming) before the guardrail's boolean result is even inspected. Left in place, to_state() would still hand back a RunState carrying a completed handoff built on guardrail-rejected input -- and Runner.run() always treats a RunState input as an already-resumed run (a plain isinstance check), so resuming it would skip the starting agent's own input guardrails entirely and continue under the delegate. Trims every affected owner back to its pre-turn length right before raising InputGuardrailTripwireTriggered, reusing the same _BlockedOutputOwnerStarts snapshot the handoff branch already takes for blocked-output handling rather than adding a second mechanism. New test confirms a tripwired turn 0 leaves state._generated_items, _session_items, and _model_responses empty via to_state() -- fails on unpatched code (asserts the handoff pair is present) and passes after. --------- Co-authored-by: Justin Beckwith <[email protected]>
1 parent 99f8d77 commit cd437f0

7 files changed

Lines changed: 637 additions & 76 deletions

File tree

‎src/agents/memory/openai_responses_compaction_session.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -431,7 +431,6 @@ async def _run_compaction_locked(
431431
if is_automatic and snapshot is None and self.max_rollback_items is not None:
432432
await self._get_all_underlying_session_items()
433433

434-
self._deferred_response_id = None
435434
logger.debug(
436435
"compact: start for %s using %s (mode=%s)",
437436
self._response_id,
@@ -490,6 +489,12 @@ async def _run_compaction_locked(
490489
)
491490
self._session_items = None if read_items is not None else output_items
492491

492+
# Clear the deferred marker only now that compaction has actually settled. Clearing it
493+
# before the fallible API call/replacement above would let a failed forced compaction
494+
# silently lose its "this must be forced" signal: a later retry recomputes `force` from
495+
# this marker, so an early clear makes the retry decline work that was still owed.
496+
self._deferred_response_id = None
497+
493498
logger.debug(
494499
"compact: done for %s (mode=%s, output=%s, candidates=%s)",
495500
self._response_id,

‎src/agents/run_internal/agent_runner_helpers.py‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -505,18 +505,16 @@ def build_interruption_result(
505505

506506

507507
def reject_unrecoverable_terminal_state(run_state: RunState | None) -> None:
508-
"""Fail closed when a previous run already produced a final output that cannot be reproduced.
508+
"""Fail closed when a previous run reached a boundary that cannot safely be resumed.
509509
510-
The marker is set once that output, its guardrails, and its terminal hooks have completed,
511-
and is cleared only once the turn is fully persisted. In between, the run owns a result no
512-
resume can settle, so resuming would repeat the model call and the lifecycle hooks for an
513-
output the caller already received. Raised before any Session, sandbox, model, tool,
514-
guardrail, or hook work so the rejection has no side effects of its own.
510+
This includes failed terminal persistence and a speculative handoff rejected by an input
511+
guardrail. Resuming would repeat completed work or bypass the failed input check. Reject
512+
before any Session, sandbox, model, tool, guardrail, or hook work.
515513
"""
516514
if run_state is not None and run_state._terminal_unrecoverable:
517515
raise UserError(
518-
"This RunState already produced a final output whose Session write did not "
519-
"complete, so it cannot be resumed. Start a new run instead."
516+
"This RunState ended at an unrecoverable boundary and cannot be resumed. "
517+
"Start a new run instead."
520518
)
521519

522520

‎src/agents/run_internal/run_loop.py‎

Lines changed: 38 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
_detach_data_redacted_error_traceback,
4444
_is_error_data_redacted,
4545
_mark_error_data_redacted,
46+
_mark_error_to_drain_stream_events,
4647
_prepare_data_redacted_error,
4748
)
4849
from ..guardrail import OutputGuardrailResult
@@ -422,6 +423,7 @@ async def _save_stream_items(
422423
response_id: str | None,
423424
update_persisted_count: bool,
424425
store: bool | None = None,
426+
resumed_write_state: RunState | None = None,
425427
) -> None:
426428
if not await _should_persist_stream_items(
427429
session=session,
@@ -437,6 +439,7 @@ async def _save_stream_items(
437439
response_id=response_id,
438440
store=store,
439441
wrapper=streamed_result.context_wrapper,
442+
resumed_write_state=resumed_write_state,
440443
)
441444
if update_persisted_count and streamed_result._state is not None:
442445
streamed_result._current_turn_persisted_item_count = (
@@ -1165,6 +1168,12 @@ async def _save_stream_items_without_count(
11651168
response_id=response_id,
11661169
update_persisted_count=False,
11671170
store=store_setting,
1171+
resumed_write_state=(
1172+
run_state
1173+
if run_state is not None
1174+
and isinstance(run_state._current_step, NextStepRunAgain)
1175+
else None
1176+
),
11681177
)
11691178

11701179
async def _save_max_turns_items(
@@ -1886,23 +1895,42 @@ def _record_max_turns_handler_output(
18861895
server_conversation_tracker.track_server_items(turn_result.model_response)
18871896

18881897
if isinstance(turn_result.next_step, NextStepHandoff):
1889-
await _save_stream_items_without_count(
1890-
turn_session_items,
1891-
turn_result.model_response.response_id,
1892-
store_setting,
1893-
)
1898+
# A failed input check makes this completed speculative turn non-resumable:
1899+
# replaying it would skip the starting agent's guardrails. Keep its executed
1900+
# call/output records coherent instead of partially rolling back history.
1901+
try:
1902+
if streamed_result._input_guardrails_task is not None:
1903+
await streamed_result._input_guardrails_task
1904+
for guardrail_result in streamed_result.input_guardrail_results:
1905+
if guardrail_result.output.tripwire_triggered:
1906+
raise InputGuardrailTripwireTriggered(guardrail_result)
1907+
except BaseException:
1908+
if run_state is not None:
1909+
run_state._terminal_unrecoverable = True
1910+
raise
18941911
current_agent = turn_result.next_step.new_agent
18951912
if run_state is not None:
18961913
run_state._current_agent = current_agent
18971914
_publish_streamed_result_agent(streamed_result, current_agent)
1898-
current_span.finish(reset_current=True)
1899-
current_span = None
1900-
should_run_agent_start_hooks = True
1915+
if streamed_result._state is not None:
1916+
streamed_result._state._current_step = NextStepRunAgain()
1917+
# Queue the agent-transition event before the fallible session append so
1918+
# stream consumers observe the transition even if the append later raises.
19011919
streamed_result._event_queue.put_nowait(
19021920
AgentUpdatedStreamEvent(new_agent=current_agent)
19031921
)
1904-
if streamed_result._state is not None:
1905-
streamed_result._state._current_step = NextStepRunAgain()
1922+
try:
1923+
await _save_stream_items_without_count(
1924+
turn_session_items,
1925+
turn_result.model_response.response_id,
1926+
store_setting,
1927+
)
1928+
except BaseException as session_persistence_error:
1929+
_mark_error_to_drain_stream_events(session_persistence_error)
1930+
raise
1931+
current_span.finish(reset_current=True)
1932+
current_span = None
1933+
should_run_agent_start_hooks = True
19061934

19071935
if await _wait_for_streamed_turn_events_and_stop_if_cancelled(streamed_result):
19081936
break

‎src/agents/run_internal/session_persistence.py‎

Lines changed: 100 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -631,6 +631,71 @@ def update_run_state_after_resume(
631631
run_state._current_step = next_step # type: ignore[assignment]
632632

633633

634+
async def _apply_post_write_compaction(
635+
session: Session,
636+
*,
637+
response_id: str | None,
638+
store: bool | None,
639+
has_local_tool_outputs: bool,
640+
wrapper: RunContextWrapper[Any] | None = None,
641+
) -> None:
642+
"""Evaluate deferred/forced Responses compaction for a settled session append.
643+
644+
Shared by the immediate-write path in ``save_result_to_session`` and the checkpoint
645+
replay path in ``resume_pending_session_write``, so a batch that only settles later
646+
(via a separate resume) still gets the same compaction decision it would have gotten
647+
had the original append succeeded inline. ``wrapper`` is the caller's raw (pre-gating)
648+
context wrapper; it is used as-is for ``run_compaction`` and re-gated here for
649+
``_defer_compaction``, mirroring the two call sites this helper replaces.
650+
"""
651+
if not response_id or not is_openai_responses_compaction_aware_session(session):
652+
return
653+
654+
if has_local_tool_outputs:
655+
defer_compaction = getattr(session, "_defer_compaction", None)
656+
if callable(defer_compaction):
657+
await _call_session_method(
658+
defer_compaction,
659+
response_id,
660+
store=store,
661+
wrapper=_get_session_wrapper(session, wrapper),
662+
)
663+
logger.debug(
664+
"skip: deferring compaction for response %s due to local tool outputs",
665+
response_id,
666+
)
667+
return
668+
669+
deferred_response_id = None
670+
get_deferred = getattr(session, "_get_deferred_compaction_response_id", None)
671+
if callable(get_deferred):
672+
deferred_response_id = get_deferred()
673+
force_compaction = deferred_response_id is not None
674+
if force_compaction:
675+
logger.debug(
676+
"compact: forcing for response %s after deferred %s",
677+
response_id,
678+
deferred_response_id,
679+
)
680+
compaction_args: OpenAIResponsesCompactionArgs = {
681+
"response_id": response_id,
682+
"force": force_compaction,
683+
}
684+
if store is not None:
685+
compaction_args["store"] = store
686+
if wrapper is not None:
687+
wrapper._session_compaction_is_automatic = True # type: ignore[attr-defined]
688+
try:
689+
await _call_session_method(
690+
session.run_compaction,
691+
compaction_args,
692+
wrapper=wrapper,
693+
)
694+
finally:
695+
if wrapper is not None:
696+
wrapper._session_compaction_is_automatic = False # type: ignore[attr-defined]
697+
698+
634699
async def save_result_to_session(
635700
session: Session | None,
636701
original_input: str | list[TResponseInputItem],
@@ -738,6 +803,10 @@ async def save_result_to_session(
738803
run_state._current_turn_persisted_item_count = already_persisted + saved_run_items_count
739804
return saved_run_items_count
740805

806+
has_local_tool_outputs = any(
807+
isinstance(item, ToolCallOutputItem | HandoffOutputItem) for item in new_items
808+
)
809+
741810
if resumed_write_state is not None:
742811
if resumed_write_state._pending_session_write is not None:
743812
raise UserError("Resolve the pending Session write before saving another batch")
@@ -748,7 +817,13 @@ async def save_result_to_session(
748817
"persisted_count": (
749818
resumed_write_state._current_turn_persisted_item_count + saved_run_items_count
750819
),
820+
"response_id": response_id,
821+
"store": store,
822+
"has_local_tool_outputs": has_local_tool_outputs,
751823
}
824+
# resume_pending_session_write() applies post-write compaction itself once the
825+
# checkpoint settles, whether that happens inline below or on a later, separate
826+
# resume -- so it is not repeated after this call returns.
752827
await resume_pending_session_write(
753828
resumed_write_state,
754829
session,
@@ -760,53 +835,14 @@ async def save_result_to_session(
760835
if run_state is not None:
761836
run_state._current_turn_persisted_item_count = already_persisted + saved_run_items_count
762837

763-
if response_id and is_openai_responses_compaction_aware_session(session):
764-
has_local_tool_outputs = any(
765-
isinstance(item, ToolCallOutputItem | HandoffOutputItem) for item in new_items
838+
if resumed_write_state is None:
839+
await _apply_post_write_compaction(
840+
session,
841+
response_id=response_id,
842+
store=store,
843+
has_local_tool_outputs=has_local_tool_outputs,
844+
wrapper=compaction_wrapper,
766845
)
767-
if has_local_tool_outputs:
768-
defer_compaction = getattr(session, "_defer_compaction", None)
769-
if callable(defer_compaction):
770-
await _call_session_method(
771-
defer_compaction,
772-
response_id,
773-
store=store,
774-
wrapper=wrapper,
775-
)
776-
logger.debug(
777-
"skip: deferring compaction for response %s due to local tool outputs",
778-
response_id,
779-
)
780-
return saved_run_items_count
781-
782-
deferred_response_id = None
783-
get_deferred = getattr(session, "_get_deferred_compaction_response_id", None)
784-
if callable(get_deferred):
785-
deferred_response_id = get_deferred()
786-
force_compaction = deferred_response_id is not None
787-
if force_compaction:
788-
logger.debug(
789-
"compact: forcing for response %s after deferred %s",
790-
response_id,
791-
deferred_response_id,
792-
)
793-
compaction_args: OpenAIResponsesCompactionArgs = {
794-
"response_id": response_id,
795-
"force": force_compaction,
796-
}
797-
if store is not None:
798-
compaction_args["store"] = store
799-
if compaction_wrapper is not None:
800-
compaction_wrapper._session_compaction_is_automatic = True # type: ignore[attr-defined]
801-
try:
802-
await _call_session_method(
803-
session.run_compaction,
804-
compaction_args,
805-
wrapper=compaction_wrapper,
806-
)
807-
finally:
808-
if compaction_wrapper is not None:
809-
compaction_wrapper._session_compaction_is_automatic = False # type: ignore[attr-defined]
810846

811847
return saved_run_items_count
812848

@@ -886,10 +922,10 @@ def digests(items: Sequence[TResponseInputItem]) -> list[str]:
886922
append = True
887923
else:
888924
expected = before + digests(pending["items"])
889-
committed_generation: int | None = None
925+
observed_generation: int | None = None
890926
get_with_generation = getattr(session, "_get_items_with_generation", None)
891927
if wrapper is not None and callable(get_with_generation):
892-
tail, committed_generation = await _call_session_method(
928+
tail, observed_generation = await _call_session_method(
893929
get_with_generation,
894930
lambda: _session_get_items(session, limit=len(expected), wrapper=wrapper),
895931
)
@@ -904,12 +940,26 @@ def digests(items: Sequence[TResponseInputItem]) -> list[str]:
904940
"Repair the original Session before resuming; do not rerun the completed tool."
905941
)
906942
append = unchanged
907-
if committed and committed_generation is not None and wrapper is not None:
908-
wrapper._session_compaction_generation = committed_generation # type: ignore[attr-defined]
943+
# The original append can advance the wrapper generation even when it fails
944+
# atomically. Reconciled unchanged history is also safe to append against;
945+
# subsequent mutations still revoke ownership through the normal generation check.
946+
if observed_generation is not None and wrapper is not None:
947+
wrapper._session_compaction_generation = observed_generation # type: ignore[attr-defined]
909948
if append:
910949
# Backends may retain or transform their input; the durable checkpoint stays detached.
911950
await _session_add_items(session, copy.deepcopy(pending["items"]), wrapper=wrapper)
912951
run_state._current_turn_persisted_item_count = pending["persisted_count"]
952+
# Keep the checkpoint until compaction also settles: if _apply_post_write_compaction
953+
# raises below, a later retry must still be able to redo just the compaction step
954+
# instead of silently losing it. The append itself is retry-safe (the reconciliation
955+
# above detects an already-committed batch and skips re-appending it).
956+
await _apply_post_write_compaction(
957+
session,
958+
response_id=pending.get("response_id"),
959+
store=pending.get("store"),
960+
has_local_tool_outputs=pending.get("has_local_tool_outputs", False),
961+
wrapper=wrapper,
962+
)
913963
run_state._pending_session_write = None
914964
finally:
915965
run_state._session_write_in_progress = False

0 commit comments

Comments
 (0)