Clarify, warn, and recover message-tool-only finals by anyech · Pull Request #80949 · openclaw/openclaw

anyech · 2026-05-12T08:47:18Z

Summary

add prompt/context guidance so directly addressed/status/completion answers in message_tool_only conversations use the message tool instead of private normal final text
add a privacy-safe diagnostic warning when a substantive normal assistant final is suppressed by message_tool_only source delivery
add a guarded runtime fallback that delivers missed message_tool_only normal finals through the normal final delivery path when no committed message-tool delivery evidence exists
keep suppressed final content out of diagnostic warnings; diagnostics record only route/session metadata plus aggregate count/text length
keep sendPolicy: deny authoritative and avoid duplicate visible posts when the message tool already delivered

Why

On message-tool-only group/channel surfaces, normal assistant final text is intentionally private. If an agent completes work but puts the user-facing status/result in the normal final instead of calling the message tool, the channel can look stalled even though the model produced a useful answer.

This PR consolidates the prompt-only guidance from #80050 with the runtime audit/recovery work here so there is one review surface for the final-visibility issue.

The fix now handles three layers:

prompt/tool guidance tells the agent to use message(action=send) for visible source replies;
if a substantive final still gets suppressed, emit an operator-facing audit warning without leaking the suppressed text;
if the runner can prove there was no committed message-tool delivery for the turn and source delivery policy allows it, mark the final payload for same-turn visible fallback delivery.

Runtime policy

The fallback is intentionally narrow:

only applies when sourceReplyDeliveryMode === "message_tool_only"
only applies when there is no committed message-tool delivery evidence (messagingToolSentTexts, media URLs, or committed targets)
does not bypass sendPolicy: deny
uses the normal final delivery path rather than channel-specific special cases
leaves existing runtime-failure notice handling (deliverDespiteSourceReplySuppression) intact

Tests

git diff --check
corepack pnpm install --frozen-lockfile --ignore-scripts
corepack pnpm exec vitest run src/auto-reply/reply/dispatch-from-config.reply-dispatch.test.ts src/auto-reply/reply/dispatch-from-config.test.ts (119 passed)
corepack pnpm exec vitest run src/auto-reply/reply/agent-runner.runreplyagent.e2e.test.ts --config test/vitest/vitest.e2e.config.ts (50 passed)
corepack pnpm exec vitest run src/auto-reply/reply/dispatch-from-config.reply-dispatch.test.ts src/auto-reply/reply/dispatch-from-config.test.ts src/auto-reply/reply/agent-runner.runreplyagent.e2e.test.ts src/auto-reply/reply/groups.test.ts src/agents/system-prompt.test.ts src/agents/tools/message-tool.test.ts src/auto-reply/reply/dispatch-acp.test.ts (339 passed in the default multi-project config; the e2e file is also covered separately by the command above)
OPENCLAW_OXLINT_SKIP_PREPARE=1 node scripts/run-oxlint.mjs --tsconfig config/tsconfig/oxlint.core.json src/auto-reply/reply/agent-runner.runreplyagent.e2e.test.ts (0 warnings / 0 errors after the lint-only follow-up commit)

Real behavior proof

Behavior or issue addressed: message_tool_only final-visibility recovery: a normal final reply that missed the message tool is either audited as suppressed without leaking text, or, when explicitly marked as the guarded fallback and policy allows it, delivered through the normal final path.
Real environment tested: disposable source-stage checkout of PR head 88708d4ce7 on Linux arm64 / Node 22. The harness used the current source dispatch runtime with an isolated session store and plugins.enabled: false; no live Gateway, live Discord/Telegram channel, channel credentials, or watched config were used.
Exact steps or command run after this patch: from the disposable checkout, run tsx message-tool-final-fallback-proof.mts, a terminal runtime harness that imports the current source dispatchReplyFromConfig, setReplyPayloadMetadata, and diagnostics event path, then exercises three message_tool_only final-delivery cases.
Evidence after fix: terminal output from that harness:

{
  "proof": "message-tool-only final fallback terminal runtime harness",
  "liveGatewayUsed": false,
  "liveChannelUsed": false,
  "sourceMode": "message_tool_only",
  "results": [
    {
      "case": "marked fallback delivers through final path",
      "queuedFinal": true,
      "sourceReplyDeliveryMode": "message_tool_only",
      "queuedCounts": { "tool": 0, "block": 0, "final": 1 },
      "finalPayloads": [
        {
          "textLen": 42,
          "hasFallbackMarker": true,
          "hasSuppressionBypassMarker": false
        }
      ],
      "warningCount": 0,
      "warnings": []
    },
    {
      "case": "unmarked final is suppressed and audited without text",
      "queuedFinal": false,
      "sourceReplyDeliveryMode": "message_tool_only",
      "queuedCounts": { "tool": 0, "block": 0, "final": 0 },
      "finalPayloads": [],
      "warningCount": 1,
      "warnings": [
        {
          "level": "warning",
          "loggerName": "auto-reply.dispatch",
          "suppressedFinalCount": 1,
          "suppressedFinalTextChars": 42,
          "sendPolicyDenied": false,
          "hasSuppressedText": false
        }
      ]
    },
    {
      "case": "sendPolicy deny still blocks marked fallback",
      "queuedFinal": false,
      "sourceReplyDeliveryMode": "message_tool_only",
      "queuedCounts": { "tool": 0, "block": 0, "final": 0 },
      "finalPayloads": [],
      "warningCount": 0,
      "warnings": []
    }
  ]
}

Observed result after fix: the marked fallback produced one queued final payload with the fallback metadata still present; the ordinary unmarked final stayed suppressed and emitted only aggregate warning metadata (suppressedFinalCount, text length, policy flag) with hasSuppressedText: false; sendPolicy: deny stayed authoritative and blocked even a marked fallback.
What was not tested: live Discord/Telegram posting was intentionally not run for this proof; no live Gateway restart, live config write, live channel credential, or external channel send was used.

Scope

This is a source/runtime delivery fix only. It does not change installed dist artifacts, live gateway config, or channel credentials.

clawsweeper · 2026-05-12T08:50:35Z

Codex review: needs real behavior proof before merge.

Summary
The PR adds message-tool-only prompt/tool guidance, suppressed-final diagnostics, and a metadata-based fallback that can deliver missed normal finals through the final reply path.

Reproducibility: yes. by source inspection: current group/channel turns resolve to message_tool_only and dispatch suppresses ordinary final payloads, so a final answer without message(action=send) stays private. I did not run a live channel repro in this read-only review.

Real behavior proof
Needs stronger real behavior proof before merge: The PR body has terminal source-harness output, but contributor action is still needed for redacted live Discord/Telegram or Gateway-to-channel proof showing fallback behavior, no duplicate posts, and no private-final leakage; after updating the PR body, ClawSweeper should re-review automatically or a maintainer can comment @clawsweeper re-review.

Next step before merge
Human maintainer review is needed because the remaining blocker is a privacy/product policy decision plus missing live channel proof, not a narrow mechanical repair.

Security
Needs attention: The patch introduces a privacy-sensitive path that can visibly post normal final text on tool-only channels.

Review findings

[P1] Gate private-final fallback behind visible-delivery policy — src/auto-reply/reply/agent-runner.ts:1727-1735

Review details

Best possible solution:

Keep the prompt guidance and privacy-safe audit, but make same-turn auto-delivery diagnostic-only unless maintainers explicitly approve a visible-delivery policy or config gate with live Discord/Telegram proof.

Do we have a high-confidence way to reproduce the issue?

Yes by source inspection: current group/channel turns resolve to message_tool_only and dispatch suppresses ordinary final payloads, so a final answer without message(action=send) stays private. I did not run a live channel repro in this read-only review.

Is this the best way to solve the issue?

No. The guidance and audit warning are narrow, but default auto-delivery of normal finals is not the safest maintainable fix without an explicit policy/config gate and live channel proof.

Full review comments:

[P1] Gate private-final fallback behind visible-delivery policy — src/auto-reply/reply/agent-runner.ts:1727-1735
runReplyAgent marks every message_tool_only final as a fallback when no message-tool evidence exists, and dispatch then posts it despite source-reply suppression. That makes normal final text visible even though the docs and maintainer policy say tool-only rooms keep normal finals private unless automatic mode is configured or the agent explicitly calls message.send; gate this behind explicit visible-delivery policy/config, or keep it diagnostic-only.
Confidence: 0.89

Overall correctness: patch is incorrect
Overall confidence: 0.88

Security concerns:

[high] Private final text can be posted visibly — src/auto-reply/reply/agent-runner.ts:1727
The fallback marker is derived from delivery mode plus absence of message-tool evidence, not from an explicit visible-response policy decision. Dispatch then treats that marker as a suppression bypass, which can leak a normal final that the current tool-only contract treats as private.
Confidence: 0.89

Acceptance criteria:

node scripts/run-vitest.mjs src/auto-reply/reply/dispatch-from-config.test.ts src/auto-reply/reply/dispatch-from-config.reply-dispatch.test.ts
node scripts/run-vitest.mjs --config test/vitest/vitest.e2e.config.ts src/auto-reply/reply/agent-runner.runreplyagent.e2e.test.ts
Redacted live Telegram or Discord proof showing fallback behavior, no duplicate post, and no private-final leakage

What I checked:

Current documented contract: Current docs say group/channel rooms default to message_tool visible replies, normal finals are not automatically posted, and operators should use a tool-call-reliable model or set visibleReplies to automatic for legacy final posting. Public docs: docs/channels/groups.md. (docs/channels/groups.md:43, 4e10969aded0)
Current delivery mode source: Current main resolves group/channel turns to message_tool_only unless configured automatic, with message-tool unavailability falling back to automatic. (src/auto-reply/reply/source-reply-delivery-mode.ts:40, 4e10969aded0)
Current suppression boundary: Current main skips final payloads under source delivery suppression unless the existing internal deliverDespiteSourceReplySuppression marker is present and sendPolicy is not denied. (src/auto-reply/reply/dispatch-from-config.ts:1599, 4e10969aded0)
Maintainer policy context: The matching Telegram maintainer note states that group/channel visible replies are policy-controlled and normal room replies stay private unless automatic mode is configured or the agent explicitly calls message.send. (.agents/maintainer-notes/telegram.md:27, 4e10969aded0)
PR fallback marking: The PR marks all message_tool_only final payloads with messageToolOnlyFinalFallback when no committed message-tool delivery evidence exists. (src/auto-reply/reply/agent-runner.ts:1727, 5a05fad80124)
PR suppression bypass: The PR treats messageToolOnlyFinalFallback as permission to deliver through the normal final path despite source-reply suppression after sendPolicy checks. (src/auto-reply/reply/dispatch-from-config.ts:1649, 5a05fad80124)

Likely related people:

scoootscooob: Commit 3c63620 introduced the group/channel message-tool-only default and changed the relevant docs, dispatch, and group prompt behavior. (role: introduced behavior; confidence: high; commits: 3c636208b0a0; files: docs/channels/groups.md, src/auto-reply/reply/groups.ts, src/auto-reply/reply/dispatch-from-config.ts)
steipete: Recent commits changed visible-reply mode handling and message-tool-only tool descriptions in the central auto-reply and agent-tool paths. (role: recent area contributor; confidence: high; commits: e1fd27fb24ae, b62166301efd, 64d4f99d2641; files: src/auto-reply/reply/source-reply-delivery-mode.ts, src/agents/tools/message-tool.ts, src/auto-reply/reply/dispatch-from-config.ts)
pashpashpash: Recent work on Codex/message-tool reply routing and structured tool replies touches the same message-tool delivery evidence surface used by this PR. (role: adjacent owner; confidence: medium; commits: 439d8edf68e2, 78eb92e62277; files: src/agents/tools/message-tool.ts, src/auto-reply/reply/dispatch-from-config.ts)
obviyus: Commit 21c33be recently changed dispatch duplicate suppression around tool-only delivery, adjacent to the final-delivery suppression boundary. (role: adjacent owner; confidence: medium; commits: 21c33bed3b54; files: src/auto-reply/reply/dispatch-from-config.ts, extensions/telegram/src/bot-message-dispatch.test.ts)

Remaining risk / open question:

The fallback can expose normal final text on channels configured to keep normal finals private.
The supplied proof does not exercise a live Discord or Telegram route, so duplicate-post and no-leak behavior are not demonstrated on the affected surface.
Maintainers still need to choose warning-only, opt-in/config-gated fallback, or default same-turn recovery.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 4e10969aded0.

anyech · 2026-05-15T08:16:08Z

I took another pass over the message_tool_only final-visibility issue after seeing the same failure shape recur in a real Discord group/thread workflow: tool/progress messages were visible, the model produced a substantive normal final answer, but no visible final message was posted because the assistant did not call the message tool.

I also checked current main and the related open PRs before opening anything new:

Clarify visible replies for message-tool-only chats #80050 improves the prompt/tool guidance, but it is intentionally prompt-only. That helps, but it did not prevent the recurrence I saw.
fix: deliver group/channel replies automatically when bot is @mentioned #75327 changes delivery mode when the bot is explicitly mentioned. That may cover some cases, but it does not cover all thread-bound or otherwise activated message_tool_only turns.
This PR (Clarify, warn, and recover message-tool-only finals #80949) is the closest existing thread. It detects/audits suppressed substantive finals while preserving the current privacy boundary.

I have a tested source-only alternative branch that implements a stricter runtime fallback instead of warning-only behavior:

branch: anyech:feat/discord-final-visibility-fallback
head: a4c61ae96a8691bb212b6f8035f7058a91937db7

Design summary:

add ReplyPayloadMetadata.messageToolOnlyFinalFallback
mark final reply payloads in runReplyAgent only when:
- sourceReplyDeliveryMode === "message_tool_only", and
- there is no committed message-tool delivery evidence for the turn
let dispatchReplyFromConfig deliver marked payloads through the normal final delivery path while still requiring !sendPolicyDenied
do not mark if the message tool already delivered, to avoid duplicate visible posts

Validation on the branch:

git diff --check
corepack pnpm exec vitest run src/auto-reply/reply/dispatch-from-config.test.ts — 110 passed
corepack pnpm exec vitest run src/auto-reply/reply/agent-runner.runreplyagent.e2e.test.ts --config test/vitest/vitest.e2e.config.ts — 50 passed

I am not opening a separate PR yet because this overlaps with the policy decision in #80949. The key decision seems to be:

keep Clarify, warn, and recover message-tool-only finals #80949 as warning-only for maximum privacy conservatism;
accept a runtime fallback for message_tool_only turns when there is no committed message-tool delivery and sendPolicy allows delivery;
make the fallback config-gated if auto-delivering suppressed final text is considered too broad as a default.

My current read is that warning-only is safer, but it still leaves the user-visible workflow stalled until an operator notices the diagnostic. The fallback branch is a concrete implementation if maintainers want the runtime to repair the visible delivery miss in the same turn rather than only audit it.

steipete · 2026-05-16T15:51:30Z

Thanks for pushing this forward and for the detailed runtime harness proof.

We landed the maintainer policy version in #82609 instead: 5759364.

That keeps the current group/channel message_tool contract strict: normal final text stays private when the model misses message(action=send), and OpenClaw now records suppressed-final diagnostics plus docs/config guidance. Operators who want legacy visible finals can set messages.groupChat.visibleReplies: "automatic".

I am closing this PR because the default same-turn auto-delivery fallback here is the part we are not accepting right now. It can expose final text on surfaces configured for tool-only visible replies, and we want that behavior to remain an explicit policy/config choice rather than an implicit recovery path.

The audit/logging/docs portion was useful and is represented by #82609. Thanks again for the careful branch and proof.

openclaw-barnacle Bot added size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 12, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 12, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 12, 2026

anyech force-pushed the fix/message-tool-only-final-audit branch from 47056e8 to bfdaff5 Compare May 15, 2026 08:27

openclaw-barnacle Bot added size: L and removed size: M labels May 15, 2026

anyech changed the title ~~Warn on suppressed message-tool-only finals~~ Warn and recover suppressed message-tool-only finals May 15, 2026

anyech changed the title ~~Warn and recover suppressed message-tool-only finals~~ Clarify, warn, and recover message-tool-only finals May 15, 2026

clawsweeper Bot added the mantis: telegram-visible-proof Mantis should capture Telegram visible proof. label May 15, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 15, 2026

anyech force-pushed the fix/message-tool-only-final-audit branch 4 times, most recently from 07c4068 to 88708d4 Compare May 15, 2026 10:11

OpenClaw Assistant and others added 2 commits May 15, 2026 17:53

Warn on suppressed message-tool-only finals

3c3739e

Guard message-tool-only final reply visibility

4bc2669

Clawsistant and others added 6 commits May 15, 2026 17:54

test(auto-reply): refresh message-tool prompt snapshots

25e7cd5

Clarify visible replies for message-tool-only chats

f683b32

Clarify message-tool-only agent prompts

c9a8c52

Clarify message-tool-only tool descriptions

b858ca7

Fix message-tool fallback test lint

1f93612

Refresh message-tool prompt snapshots after rebase

5a05fad

anyech force-pushed the fix/message-tool-only-final-audit branch from 88708d4 to 5a05fad Compare May 15, 2026 18:17

steipete closed this May 16, 2026

anyech mentioned this pull request May 20, 2026

fix(discord): guard message-tool-only final delivery #84485

Closed

clawsweeper Bot mentioned this pull request Jun 21, 2026

fix: recover stranded replies in message_tool_only mode (#85714) #88992

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Clarify, warn, and recover message-tool-only finals#80949

Clarify, warn, and recover message-tool-only finals#80949
anyech wants to merge 8 commits into
openclaw:mainfrom
anyech:fix/message-tool-only-final-audit

anyech commented May 12, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented May 12, 2026 •

edited

Loading

Uh oh!

anyech commented May 15, 2026

Uh oh!

steipete commented May 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

anyech commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Runtime policy

Tests

Real behavior proof

Scope

Uh oh!

clawsweeper Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anyech commented May 15, 2026

Uh oh!

steipete commented May 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anyech commented May 12, 2026 •

edited

Loading

clawsweeper Bot commented May 12, 2026 •

edited

Loading