Skip to content

fix(core): keep completed tool guardrail results when a resumed append fails - #4651

Closed
ayaangazali wants to merge 2 commits into
openai:mainfrom
ayaangazali:fix/interruption-tool-guardrail-results
Closed

ayaangazali wants to merge 2 commits into
openai:mainfrom
ayaangazali:fix/interruption-tool-guardrail-results

Conversation

@ayaangazali

Copy link
Copy Markdown
Contributor

Summary

On a resumed turn that resolves one approval while another stays pending, the run publishes that turn's tool guardrail results only inside the NextStepInterruption branch, and that branch sits after the resumed Session append:

save_resumed_turn_items(...)            # can raise
...
if isinstance(turn_result.next_step, NextStepInterruption):
    tool_input_guardrail_results.extend(turn_result.tool_input_guardrail_results)
    tool_output_guardrail_results.extend(turn_result.tool_output_guardrail_results)

When that append raises, the approved tool has already executed and both its input and output guardrails have already returned, but nothing carries their results into RunState. The run-level lists die with the raised call, and RunState only ever receives guardrail results from a built result (result.py), which never happens on this path. Nothing republishes them later either, because the recovery resume does not re-execute the tool. So the results are gone permanently: absent from the checkpoint, from its serialization, and from the final RunResult.

Measured with an allow-only input guardrail and an allow-only output guardrail on the approved tool, across sync and streamed and across a live state and a JSON round trip. ran is how many times each guardrail actually executed:

before   ran=(1,1)   state=(0,0)  serialized=(0,0)  final=(0,0)     0/4 rows retain them
after    ran=(1,1)   state=(1,1)  serialized=(1,1)  final=(1,1)     4/4

The change publishes them next to the other resumed state updates, before the append, in both run.py and run_loop.py. Both paths already seed their run-level accumulators from RunState once at run start, before the turn loop, so writing the turn's results into state mid-turn cannot double count. Confirmed on the clean path, which still reports (1, 1) and not (2, 2) in both sync and streamed.

This is the second defect reported in #4646. It is independent of #4650: different root cause, different files, and the two diffs do not touch the same functions. #4650 makes the durable Session recover the dropped append; this one makes the completed guardrail results survive the same failure. Either can land without the other.

Test plan

tests/test_run_impl_resume_paths.py::test_failed_resumed_append_keeps_completed_tool_guardrail_results, parametrized over streamed and non streamed and over a live state and a JSON round trip. It pauses on two approval-gated calls in one response, approves only the first, fails the resumed append, then asserts the checkpoint and its serialization each retain one input and one output guardrail result, and that the final result still reports them after the second approval is rejected.

Verified it fails without the source change by reverting src/ and rerunning: all four cases fail.

.agents/skills/code-change-verification/scripts/run.sh passes end to end: format, lint, typecheck and the full test suite.

Issue number

Partially addresses #4646

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

Second half of the same issue, split out because the root cause is a different one and I did not want to bury two fixes in one diff. I'm a freshman in college and still learning this part of the runner, so the thing I'd most like checked is my claim that publishing before the append cannot double count. I reasoned it from the accumulators being seeded once at run start and then verified the clean path stays at one result each, but you know the ordering here far better than I do.

…d fails

On a resumed turn that leaves another approval pending, the run publishes the
turn's tool guardrail results only inside the NextStepInterruption branch,
which sits after the resumed Session append. When that append raises, the
approved tool has already executed and both guardrails have already completed,
but their results never reach RunState, so they are absent from the checkpoint,
from its serialization, and from every later result.

Publish them alongside the other resumed state updates, before the append, in
both the streaming and non-streaming paths.
Copilot AI lite review requested due to automatic review settings August 25, 2026 06:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ed6c3ee4f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run.py Outdated
Comment on lines +1141 to +1144
run_state._tool_input_guardrail_results = [
*tool_input_guardrail_results,
*turn_result.tool_input_guardrail_results,
]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid duplicating pre-approval guardrail results on retry

When pre_approval_tool_input_guardrails=True, the interrupted state already contains the guardrail's pre-approval result, and the required resume-time recheck produces another result. A normal NextStepRunAgain resume intentionally excludes that recheck from run-wide accounting, but this unconditional concatenation stores both results if the resumed Session append fails; retrying from the live or serialized state then reports two input-guardrail results instead of the established count of one. Preserve the existing run-again accounting when checkpointing this failure path.

AGENTS.md reference: AGENTS.md:L147-L149

Useful? React with 👍 / 👎.

Comment thread src/agents/run_internal/run_loop.py Outdated
Comment on lines +1360 to +1366
run_state._tool_input_guardrail_results = [
*accepted_tool_input_guardrail_results,
*turn_result.tool_input_guardrail_results,
]
run_state._tool_output_guardrail_results = [
*accepted_tool_output_guardrail_results,
*turn_result.tool_output_guardrail_results,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Copy resumed results into the streamed checkpoint

When streaming resumes the sole pending approval, the step becomes NextStepRunAgain, so the conditional below skips _accumulate_tool_guardrail_results and these assignments update only the source RunState. If the resumed Session append then raises, the supported recovery path of calling RunResultStreaming.to_state() copies the stale streamed-result lists rather than these updated lists; the detached checkpoint therefore loses both completed guardrail results, and a retry cannot report them. Publish the results to the streamed checkpoint before the append while retaining the accepted-list deduplication rules.

AGENTS.md reference: AGENTS.md:L147-L149

Useful? React with 👍 / 👎.

…ansition

The previous commit published the resumed turn's guardrail results on every
resumed turn. On a sole pending approval the step is NextStepRunAgain, and the
streamed recovery path rebuilds its checkpoint from RunResultStreaming, so the
source RunState and the detached checkpoint disagreed after a failed append.

Guard both publications on NextStepInterruption. Run-again accounting is left
exactly as it was, and the renewed-interruption transition still matches the
clean path.
@ayaangazali

Copy link
Copy Markdown
Contributor Author

Thanks, the streaming one was real and I have fixed it. The pre-approval one I could not reproduce, and the measurement disagrees with the premise, so I would rather show the numbers than quietly change the code.

Copy resumed results into the streamed checkpoint: confirmed, fixed.

Reproduced exactly as described. Streaming a sole pending approval makes the step NextStepRunAgain, and my publication ran on every resumed turn, so the source RunState and the detached RunResultStreaming.to_state() checkpoint disagreed after a failed append. That divergence did not exist before my change:

                 step                    source RunState   detached to_state()
main   tools=1   next_step_run_again     (0,0)             (0,0)   consistent
mine   tools=1   next_step_run_again     (1,1)             (0,0)   I introduced this

Rather than teach the streaming path to publish into both owners, I narrowed the change: both publications are now guarded on NextStepInterruption. Run-again accounting is byte for byte what it was on main, and the two checkpoints agree again:

tools=2  next_step_interruption   source (1,1)   detached (1,1)   MATCH   (the reported bug, fixed)
tools=1  next_step_run_again      source (0,0)   detached (0,0)   MATCH   (unchanged from main)

That is a smaller diff than the one you reviewed, not a larger one, and it removes the second owner problem instead of adding a branch for it.

Avoid duplicating pre-approval guardrail results on retry: I do not think this one holds.

The premise is that the established count is one. On main, with pre_approval_tool_input_guardrails=True and a renewed interruption, the guardrail genuinely runs twice and the clean path already stores both results:

main, pre_approval=True, renewed interruption, clean path:   ran=(2,1)   state=(2,1)

So two is the established count for that configuration, not one. My change makes the failure path match that clean path rather than diverge from it:

                              clean     failed append
main   pre_approval=False     (1,1)     (0,0)
main   pre_approval=True      (2,1)     (1,0)
mine   pre_approval=False     (1,1)     (1,1)
mine   pre_approval=True      (2,1)     (2,1)

Both modes now agree with their own clean path, which is the invariant I was aiming for. And after the narrowing above, the NextStepRunAgain resume you were protecting is not touched at all, so the run-again exclusion is preserved by construction rather than by my getting the arithmetic right.

Happy to be shown wrong on the second one if the intended contract really is one result per call regardless of the pre-approval recheck. In that case the clean path looks like it has the same problem, and I would rather fix that separately than special case the failure path.

@seratch

seratch commented Aug 26, 2026

Copy link
Copy Markdown
Member

Thanks for the careful work on the second part of #4646. #4654 now covers the same remaining guardrail-result publication gap after #4650, but places the durable update immediately at the renewed-interruption persistence boundary and includes the stronger recovery matrix: streaming and non-streaming, live and JSON state, both ambiguous Session outcomes, and pre-approval guardrail accounting.

To keep one implementation and one regression suite for this invariant, we are going to proceed with #4654 and close this PR as superseded. The underlying issue and contribution are valid; this is only a choice between overlapping implementation paths.

@seratch seratch closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants