Skip to content

fix: prevent hidden pre-tool text from confusing channel users#102173

Open
849261680 wants to merge 2 commits into
openclaw:mainfrom
849261680:fix/101793-tooluse-undelivered-note
Open

fix: prevent hidden pre-tool text from confusing channel users#102173
849261680 wants to merge 2 commits into
openclaw:mainfrom
849261680:fix/101793-tooluse-undelivered-note

Conversation

@849261680

@849261680 849261680 commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #101793

What Problem This Solves

Fixes an issue where users on message channels could miss assistant text that was emitted before a tool call, while the next model turn still replayed that text as if it had been shown to the user.

Why This Change Was Made

The change keeps commentary-phase pre-tool text private from channel delivery, but marks that text as undelivered in provider replay so the next model turn does not assume the user saw or answered it. The replay marker preserves the original Responses replay id when present, drops the commentary phase metadata, escapes internal context delimiters, and leaves the tool call/tool result sequence intact.

Non-goal: this does not change Signal/Telegram/Desktop delivery policy or mutate stored transcript history.

User Impact

Agents no longer continue from suppressed pre-tool narration as though it was visible to the user. If the suppressed text matters after the tool result, the agent is prompted to ask again or send it in a user-visible final reply.

Evidence

  • Provider replay probe for stop + toolCall + commentary text passed with current source:
{
  "probe": "provider-replay-stop-plus-toolcall-commentary",
  "inputStopReason": "stop",
  "inputTextPhase": "commentary",
  "outputTextIncludesUndeliveredMarker": true,
  "outputTextInstructsNotSeen": true,
  "originalCommentaryRetainedForModel": true,
  "outputTextSignature": "{\"v\":1,\"id\":\"msg_live_probe\"}",
  "outputToolCallPreserved": true,
  "outputToolResultPreserved": true,
  "persistedInputUnchanged": true
}
  • OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=300000 node scripts/run-vitest.mjs src/agents/embedded-agent-runner/replay-history.test.ts passed: 1 file, 31 tests.
  • pnpm exec oxfmt --check src/agents/embedded-agent-runner/replay-history.ts src/agents/embedded-agent-runner/replay-history.test.ts passed.
  • pnpm exec oxlint --type-aware src/agents/embedded-agent-runner/replay-history.ts src/agents/embedded-agent-runner/replay-history.test.ts passed.
  • git diff --check passed.
  • .agents/skills/autoreview/scripts/autoreview --mode local passed with no accepted/actionable findings.

Agent transcript note: a local candidate session was found, but no redacted transcript section is included because explicit approval is required before adding local session logs to a public PR body.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jul 8, 2026
@clawsweeper

clawsweeper Bot commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 9:16 AM ET / 13:16 UTC.

Summary
The PR rewrites commentary-phase assistant text in tool-call replay as an internal undelivered-context marker and adds replay-history regression coverage for toolUse and stop-labeled tool-call turns.

PR surface: Source +69, Tests +83. Total +152 across 2 files.

Reproducibility: yes. at source level: current Anthropic transport tags text-plus-tool turns as commentary, and the subscriber suppresses commentary-phase output from visible channel replies. I did not run a live Signal replay in this read-only review.

Review metrics: 1 noteworthy metric.

  • Replay normalization surface: 1 cross-provider normalizer changed. The changed helper runs before provider-owned replay hooks, so this policy affects model-visible session state across routes.

Stored data model
Persistent data-model change detected: serialized state: src/agents/embedded-agent-runner/replay-history.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #101793
Summary: This PR is the candidate fix for the linked Signal/Anthropic hidden pre-tool text issue; older and broader reports are useful context but do not supersede the linked issue.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Get maintainer confirmation that replay-marker semantics are the intended delivery-policy direction before merge.
  • Run normal exact-head merge validation after the branch is refreshed against current main.

Risk before merge

  • [P1] Merging changes model-visible replay for existing commentary-plus-tool turns: suppressed text becomes an internal undelivered marker instead of ordinary assistant text.
  • [P1] The contributor proof demonstrates provider replay behavior, but it is not a full Signal client round trip for the linked P1 channel workflow.

Maintainer options:

  1. Accept the Replay Marker Contract (recommended)
    Proceed after maintainer sign-off and exact-head validation because the patch now covers tool-call-containing turns and has replay-level proof.
  2. Ask for a Channel Round Trip
    Require a redacted Signal or comparable channel transcript/client comparison before merge if replay-level proof is not enough for this workflow.
  3. Pause for a Different Delivery Policy
    Hold this PR if maintainers prefer delivering selected pre-tool text or requiring explicit message-tool sends instead of replay markers.

Next step before merge

  • [P2] No narrow automation repair remains; the next step is maintainer acceptance of the replay-marker policy and normal exact-head validation.

Maintainer decision needed

  • Question: Should this PR's undelivered replay-marker semantics be the accepted delivery contract for commentary text suppressed from messaging channels before tool calls?
  • Rationale: The patch is a bounded bug fix, but it intentionally changes what later model turns see for suppressed channel text, and the linked issue explicitly presents delivery, private replay markers, or explicit send semantics as policy choices.
  • Likely owner: vincentkoc — This is the strongest current-main routing signal for replay normalization and Anthropic commentary behavior among the likely owners found.
  • Options:
    • Accept Replay Markers (recommended): Land this approach after normal exact-head CI because it preserves commentary suppression while preventing phantom shared context.
    • Require Channel Delivery: Ask for a different fix that safely flushes only user-visible pre-tool text without regressing commentary leak suppression.
    • Require Explicit Sends: Keep bare pre-tool text private and document that channel-visible confirmations must use the message tool or another explicit send path.

Security
Cleared: The diff changes replay text normalization and focused tests only; I found no concrete security or supply-chain regression.

Review details

Best possible solution:

Land the replay-marker approach after maintainer acceptance and exact-head validation, while keeping broader explicit-send and commentary-leak policy work with #25592.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current Anthropic transport tags text-plus-tool turns as commentary, and the subscriber suppresses commentary-phase output from visible channel replies. I did not run a live Signal replay in this read-only review.

Is this the best way to solve the issue?

Yes, if maintainers accept the replay-marker policy. The latest predicate mirrors the tool-call-containing invariant and avoids the broader, leak-prone alternative of delivering all pre-tool text.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e25fa79c5d19.

Label changes

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body and follow-up include copied redacted provider replay output showing the after-fix marker behavior; this is sufficient replay proof, though not a full Signal client round trip.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body and follow-up include copied redacted provider replay output showing the after-fix marker behavior; this is sufficient replay proof, though not a full Signal client round trip.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P1: The PR targets a linked channel workflow where hidden assistant text caused later user replies to be interpreted against unseen context.
  • merge-risk: 🚨 session-state: The diff changes provider-visible replay history for commentary-plus-tool turns, so merge behavior can alter later model/session context.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body and follow-up include copied redacted provider replay output showing the after-fix marker behavior; this is sufficient replay proof, though not a full Signal client round trip.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body and follow-up include copied redacted provider replay output showing the after-fix marker behavior; this is sufficient replay proof, though not a full Signal client round trip.
Evidence reviewed

PR surface:

Source +69, Tests +83. Total +152 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 69 0 +69
Tests 1 83 0 +83
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 152 0 +152

What I checked:

Likely related people:

  • vincentkoc: Recent history includes replay and Anthropic transport work touching the same agent-runtime files and release stabilization around the affected surface. (role: recent area contributor; confidence: medium; commits: e085fa1a3ffd; files: src/agents/anthropic-transport-stream.ts, src/agents/embedded-agent-runner/replay-history.ts)
  • steipete: Recent main history touched the Anthropic transport path, and earlier commentary suppression work appears in adjacent agent payload/replay history. (role: recent adjacent contributor; confidence: medium; commits: 4bf70be01a21, 9eed092baa50; files: src/agents/anthropic-transport-stream.ts, src/agents/pi-embedded-runner/run/payloads.ts)
  • Shakker: Git history search shows a cluster of replay and signed-thinking sanitization commits in the agent replay area, making this a plausible secondary routing candidate for replay invariants. (role: replay-history contributor; confidence: low; commits: 44afe200e6c8, b6faa6945123; files: src/agents/embedded-agent-runner/replay-history.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (3 earlier review cycles)
  • reviewed 2026-07-08T12:37:32.400Z sha 448f0d1 :: needs real behavior proof before merge. :: [P2] Cover tool-call turns mislabeled as stop
  • reviewed 2026-07-08T12:45:04.385Z sha 448f0d1 :: needs real behavior proof before merge. :: [P2] Cover tool-call turns mislabeled as stop
  • reviewed 2026-07-08T13:05:32.883Z sha 94edd68 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 8, 2026
@849261680

849261680 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Updated after the review:

  • Fixed the predicate to mark commentary as undelivered for assistant replay content that contains a tool-call block, including stopReason: "stop" routes.
  • Added a regression test for stop + toolCall + commentary replay.
  • Updated the PR body with a redacted provider replay probe showing the marker is inserted, the original text is retained for model context, the replay id is preserved without the commentary phase, the tool call/result are preserved, and persisted input remains unchanged.

Latest local evidence:

  • OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=300000 node scripts/run-vitest.mjs src/agents/embedded-agent-runner/replay-history.test.ts passed: 1 file, 31 tests.
  • pnpm exec oxfmt --check src/agents/embedded-agent-runner/replay-history.ts src/agents/embedded-agent-runner/replay-history.test.ts passed.
  • pnpm exec oxlint --type-aware src/agents/embedded-agent-runner/replay-history.ts src/agents/embedded-agent-runner/replay-history.test.ts passed.
  • git diff --check passed.
  • .agents/skills/autoreview/scripts/autoreview --mode local passed with no accepted/actionable findings.

@clawsweeper

clawsweeper Bot commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 8, 2026
This was referenced Jul 8, 2026
Open
Closed
@openclaw-barnacle

openclaw-barnacle Bot commented Jul 23, 2026

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

agents Agent runtime and tooling merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S stale Marked as stale due to inactivity status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Assistant text preceding a tool call in the same turn is silently dropped on Signal channel (Anthropic/Sonnet, non-streaming)

1 participant

@849261680