fix(agents): retry empty post-tool final turns#93073
Conversation
|
Codex review: needs real behavior proof before merge. Reviewed June 15, 2026, 3:05 AM ET / 07:05 UTC. Summary PR surface: Source +221, Tests +486. Total +707 across 30 files. Reproducibility: yes. source-reproducible: current main can treat aggregate pre-tool narration as non-empty before considering the latest empty post-tool assistant message. Review metrics: 2 noteworthy metrics.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Security Review findings
Review detailsBest possible solution: Resolve whether replay-safe tool metadata is an intentional public plugin/SDK contract, document and test that contract if kept, and require maintainer acceptance or real logs for the exact empty-final recovery before merge. Do we have a high-confidence way to reproduce the issue? Yes, source-reproducible: current main can treat aggregate pre-tool narration as non-empty before considering the latest empty post-tool assistant message. Is this the best way to solve the issue? No for merge readiness: the typed-state fix is the right layer versus the superseded broad fallback PR, but the public plugin/SDK replay contract and proof gap still need maintainer resolution. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 1c30bb8ce639. Label changesLabel justifications:
Evidence reviewedPR surface: Source +221, Tests +486. Total +707 across 30 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
@clawsweeper re-review Addressed the fail-closed finding on Proof: focused safety matrix |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
92502cd to
3c55d30
Compare
|
@clawsweeper re-review Current head Exact-head proof: Blacksmith Testbox-through-Crabbox |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
Recover assistant turns that complete tool work without producing a visible final answer, while preserving intentional silent replies. Use concrete tool-instance replay safety across embedded, Codex, and Copilot runtimes so unknown, mutating, async-started, and durable recall operations fail closed. Preserve genuine empty Codex final items without promoting commentary or tool-progress echoes. Supersedes openclaw#90872. Thanks @fuller-stack-dev. Co-authored-by: fuller-stack-dev <[email protected]>
Recover assistant turns that complete tool work without producing a visible final answer, while preserving intentional silent replies. Use concrete tool-instance replay safety across embedded, Codex, and Copilot runtimes so unknown, mutating, async-started, and durable recall operations fail closed. Preserve genuine empty Codex final items without promoting commentary or tool-progress echoes. Supersedes openclaw#90872. Thanks @fuller-stack-dev. Co-authored-by: fuller-stack-dev <[email protected]>
Recover assistant turns that complete tool work without producing a visible final answer, while preserving intentional silent replies. Use concrete tool-instance replay safety across embedded, Codex, and Copilot runtimes so unknown, mutating, async-started, and durable recall operations fail closed. Preserve genuine empty Codex final items without promoting commentary or tool-progress echoes. Supersedes openclaw#90872. Thanks @fuller-stack-dev. Co-authored-by: fuller-stack-dev <[email protected]>
Recover assistant turns that complete tool work without producing a visible final answer, while preserving intentional silent replies. Use concrete tool-instance replay safety across embedded, Codex, and Copilot runtimes so unknown, mutating, async-started, and durable recall operations fail closed. Preserve genuine empty Codex final items without promoting commentary or tool-progress echoes. Supersedes openclaw#90872. Thanks @fuller-stack-dev. Co-authored-by: fuller-stack-dev <[email protected]>
Summary
Success means a replay-safe
checking... -> tool/search -> empty final assistanttrajectory receives one bounded continuation, while visible final answers, explicit silence, terminal media/delivery, async work, and side-effectful attempts retain their existing behavior.Linked context
Supersedes #90872.
That PR established a real channel-facing problem, but its current head is 75 files and +15,999/-1,907 with public Plugin SDK/core tool-contract changes. The maintainer decision was explicit: the 15k LOC approach is not landing.
Root cause and best-fix rationale
assistantTextsandpayloadCountaggregate visible text from the whole attempt. If the model says it is checking something before a tool call, that narration remains non-empty even when the latest post-tool assistant message is a typed emptystopturn. The existing empty-response recovery therefore never runs and the channel sees narration followed by silence.The tool result itself is valid model input, including sparse values such as
0. Parsing, rewriting, or surfacing raw tool results would move responsibility to the wrong boundary and could expose unsafe output. The narrow invariant is:Each harness replay metadata still decides whether the one continuation is safe. The embedded/Pi replay-metadata builder now positively classifies known replay-safe tools and treats unknown/action-dependent tool names as potential side effects. If replay is unsafe, it does not continue automatically and the existing verify-before-retrying warning is used; if a safe retry is exhausted, the existing incomplete-turn warning is used.
Other harnesses inspected
Source audit used exact heads:
d683a581b747stopReasonandtoolCallblocks.85e278b728f5cdc1d592df7fneeds_follow_up; empty finalized agent messages becomeNone; the turn loop continues or ends from typed state (codex-rs/core/src/session/turn.rs,codex-rs/core/src/stream_events_utils.rs).9e5599c323f1complete_tasktool and treats stopping without it as a protocol violation.9d166ecee976final_outputtool.2a14e8957d75tool_calls. After a no-tool empty response with a recenttoolrole, it inserts one synthetic continuation nudge, then uses bounded generic empty-response retries/fallback; finalization strips recovery scaffolding and surfaces an explanation. Its regex in this branch recognizes provider<think>markup, not natural-language completion.fb97af83f06dThe common pattern is typed continuation/completion state, with explicit completion tools for strict workflows. None of these findings support language-regex classification or generic raw-result fallback for a normal chat turn.
Real behavior proof
xai/grok-composer-2.5-fast; the original Discord delivery path was not used.f1b8827d20c8f1d648cdb1d4034120a67d1ff3e7with only the original regression tests applied; the expanded post-tool safety matrix and full owner test on pre-maintainer head92502cd8b4c3ce16c8ec493bf221512c7b36a656, plus exact-head focused Testbox proof on current head3c55d30f9ca7223b3068fe640a5f0a3463269f3f; plus three boundedopenclaw agent --local --model xai/grok-composer-2.5-fastturns covering healthy read-only search, terminal tool failure, and narration without a tool call.6failures across the two Vitest projects): retry resolver returnednull, exhausted handling returnednull, and the full runner stopped after one attempt. Pre-maintainer-head focused safety coverage passes20/20across the two projects, including known read-only recovery, mixed known-plus-unknown fail-closed behavior, and the side-effect-aware terminal warning. The full owner test file passes150/150.web_search,memory_search,x_search, andtool_describeremain eligible. An unclassified tool receives no automatic retry, recordsreplaySafe: false, and surfaces the verify-before-retrying warning. Live Composer completed one successful read-onlyx_searchround withCOMPOSER_FINAL_OKand no unnecessary retry. A denied terminal tool attempt was marked replay-invalid and was not replayed.x_searchround, so the exact model-generated failure was not reproduced on demand outside the deterministic captured trajectory; broad cross-channel E2E was not run.web_search; that is a separate planning-only failure and does not prove this patch. The old PR's live after-fix output validated its discarded broad fallback approach, not this narrow patch, so it is retained only as provenance.0, then emitted no final user-visible assistant reply.Tests and validation
f1b8827d20c8f1d648cdb1d4034120a67d1ff3e7+ original regression tests ->6 failedacross two projects (3logical regressions)node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run.incomplete-turn.test.ts --reporter=dot -t "known read-only|unclassified tools|unclassified tools as replay-invalid|pre-tool narration even when silence is allowed|post-tool exact NO_REPLY"->20 passedacross two projectsnode scripts/run-vitest.mjs src/agents/embedded-agent-runner/run.incomplete-turn.test.ts --reporter=dot->150 passednode scripts/run-vitest.mjs src/agents/tool-mutation.test.ts --reporter=dot->48 passed3c55d30f9ca7223b3068fe640a5f0a3463269f3f; Blacksmith Testbox-through-Crabboxtbx_01kv4v9gbv92mnv435bsqdvn9p(run) -> owner test150/150, mutation classification48/48xai/grok-composer-2.5-fast+ one successful read-onlyx_search-> finalCOMPOSER_FINAL_OK, no extra retryreplayInvalid: true, no replaynode scripts/run-vitest.mjs src/auto-reply/reply/agent-runner-payloads.test.ts --reporter=dot -t "does not dedupe text for cross-target messaging sends"-> passed (1/1); the earlier PR CI failure was the same unrelated test timing out after 127 seconds and could not be rerun from contributor permissions/workflow statepnpm format:check src/agents/embedded-agent-runner/run/incomplete-turn.ts src/agents/embedded-agent-runner/run.incomplete-turn.test.ts-> passedpnpm changed:lanes --json->core,coreTestspnpm check:changed-> passed on pre-maintainer head Blacksmith Testboxtbx_01kv41nef5fjxznf9p64z6yhct(run)3c55d30f9ca7223b3068fe640a5f0a3463269f3f-> clean, no accepted/actionable findingsgit diff --check-> clean+56/-19, tests+198/-11, two filesThe original regression fails on the exact base before the production change. The final head fixes it while failing closed for unclassified tools and preserving recovery for the known read-only tool surface.
Risk checklist
Did user-visible behavior change?
YesDid config, environment, or migration behavior change?
NoDid security, auth, secrets, network, or tool execution behavior change?
Yes: one existing bounded continuation can now run for this previously missed typed terminal state.Highest-risk area: retrying after tool activity could duplicate work or replace already-delivered terminal output.
Mitigation: the continuation remains limited to one attempt and is blocked by harness replay metadata, terminal output, committed delivery, accepted spawn, error, abort, timeout, and explicit-silence guards. The embedded/Pi replay builder now fails closed for unclassified or action-dependent tool names; those attempts surface a verify-before-retrying warning. The patch does not surface raw tool results.
Current review state
Ready for maintainer review at current head
3c55d30f9ca7223b3068fe640a5f0a3463269f3f. Exact-head focused Testbox proof and trusted autoreview are green; the prior exact-base reproduction, remote changed gate, and bounded live Composer safety/healthy-path checks remain recorded above. Current-head GitHub CI is running without failures observed at this refresh. The remaining proof gap is forcing the original successful-tool-then-empty-final Composer failure on the private Discord/model stack.