Skip to content

fix(agent-runner): deliver assistant text as fallback when model skips message tool#84328

Closed
billehunt wants to merge 1 commit into
openclaw:mainfrom
billehunt:fix/message-tool-fallback-delivery
Closed

fix(agent-runner): deliver assistant text as fallback when model skips message tool#84328
billehunt wants to merge 1 commit into
openclaw:mainfrom
billehunt:fix/message-tool-fallback-delivery

Conversation

@billehunt

@billehunt billehunt commented May 19, 2026

Copy link
Copy Markdown

Summary

Fixes #84327

When sourceReplyDeliveryMode is "message_tool_only" (the default for Telegram group/channel chats), model responses were silently dropped if the model generated text without calling the message tool. This marks those payloads for fallback delivery so the user always sees the response.

Problem

The dispatch layer in dispatch-from-config.ts suppresses all reply payloads that lack deliverDespiteSourceReplySuppression metadata when suppressDelivery is true. Payloads from the message tool get this flag via markReplyPayloadForSourceSuppressionDelivery, but plain assistant text payloads from buildEmbeddedRunPayloads never do.

When the model intermittently skips calling the message tool (observed after long idle periods when the Anthropic prompt cache goes cold), the response text is generated but silently discarded at the dispatch layer.

Fix

In agent-runner.ts, after the agent run completes: if sourceReplyDeliveryMode is "message_tool_only", the model produced text payloads, and no side-effect delivery occurred (no message tool call, no block streaming, no cron adds), mark those payloads with deliverDespiteSourceReplySuppression so they bypass the dispatch suppression.

Uses the existing hasSuccessfulSideEffectDelivery helper so the fallback only activates when no other delivery path succeeded.

Real behavior proof

Tested on a live Telegram bot (OpenClaw v2026.5.12, claude-opus-4-7 via Anthropic API, group chat with queue mode steer).

Before fix — three consecutive turns with trajectory evidence:

Time (UTC) outputTokens toolMetas didSend assistantTexts messagingToolSentTexts
20:16 196 [] false Full relevant response about meal planning []
20:17 96 [] false "Hey — did you see the lunch rec?" []
20:19 56 [] false Shortened repeat of lunch recommendation []

All three responses were silently discarded. User saw nothing.

After fix — deployed the fix and restarted gateway:

Time (UTC) outputTokens toolMetas didSend assistantTexts delivered?
21:19 11 [] false "Pong. Here." Yes — delivered to Telegram

Same pattern (model skipped message tool), but the fallback delivery kicked in and the user received the response.

What was not tested: Block streaming interaction (block streaming was enabled but the model didn't stream in the broken turns). The fix checks hasSuccessfulSideEffectDelivery which covers block streaming, so there should be no double-delivery when block streaming succeeds.

Verification

pnpm test src/auto-reply/reply/agent-runner-payloads.test.ts     # 36 passed
pnpm test src/auto-reply/reply/agent-runner-execution.test.ts    # 72 passed
pnpm test src/auto-reply/reply/agent-runner-direct-runtime-config.test.ts  # 5 passed
pnpm test src/auto-reply/reply/agent-runner.misc.runreplyagent.test.ts     # 38 passed
pnpm test src/auto-reply/reply/dispatch-from-config.test.ts      # 107 passed
pnpm test src/auto-reply/reply/source-reply-delivery-mode.test.ts # 17 passed
pnpm build  # clean

🤖 AI-assisted: diagnosis and fix authored with Claude Code (Opus 4.6). Human confirmed the fix works via live Telegram testing.

@billehunt
fix(agent-runner): deliver assistant text as fallback when model skip…
4c5aaf7 
…s message tool

When sourceReplyDeliveryMode is message_tool_only (default for Telegram
group chats), model responses were silently dropped if the model generated
text without calling the message tool. Mark those payloads for fallback
delivery so the user always sees the response.
@openclaw-barnacle openclaw-barnacle Bot added size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 19, 2026
@clawsweeper

clawsweeper Bot commented May 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Workflow note: Future ClawSweeper reviews update this same comment in place.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Summary
The PR adds a fallback in src/auto-reply/reply/agent-runner.ts that marks final assistant text payloads for source-suppression delivery when message_tool_only mode is active and no side-effect delivery succeeded.

Reproducibility: yes. from source inspection and PR evidence, but I did not run a live Telegram repro. Current main suppresses unmarked final replies in message_tool_only mode, and the PR body provides live before/after Telegram trajectory data.

PR rating
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Summary: Credible live proof and a small targeted patch make this a normal good PR, with residual risk mainly from missing focused regression coverage.

Rank-up moves:

  • Add a focused regression test if maintainers want stronger automated coverage before merge.
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

PR egg
✨ Hatched: 🥚 common Tiny Shellbean

       /\  .---.  /\         
      /  \/     \/  \        
     /   ( -   - )   \       
    |       ._.       |      
    |   /|  ===  |\   |      
     \  \|______/|/  /       
      '._  `--'  _.'         
         '-.__.-'            
       _/|_|  |_|\_          
      /__|      |__\         
       .-----------.         
      '-------------'

Rarity: 🥚 common.
Trait: keeps receipts.
Image traits: location green-check meadow; accessory CI status badge; palette seafoam, black, and opal; mood calm; pose holding its accessory up for inspection; shell starlit enamel shell; lighting soft underwater shimmer; background miniature CI buoys.
How to hatch it: once this PR reaches status: 👀 ready for maintainer look or status: 🚀 automerge armed, the PR author or a maintainer can comment @clawsweeper hatch to turn this ASCII egg into its generated creature image.
Share on X: post this hatch
Copy: My PR egg hatched a 🥚 common Tiny Shellbean in ClawSweeper.

What is this egg doing here?
  • Eggs appear after the PR passes real-behavior proof. It is here for vibes, not verdicts: it does not change labels, ratings, merge decisions, or automation.
  • The shell reacts to review momentum: open follow-up work warms it up, re-review makes it wobble, and a clean final review lets it hatch.
  • Hatchable usually means sufficient real-behavior proof, no blocking P0/P1/P2 findings, no security attention needed, and clean correctness.
  • The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
  • Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

Real behavior proof
Sufficient (live_output): The PR body includes live Telegram before/after trajectory evidence showing assistant text dropped before the patch and delivered after the fallback was deployed.

Mantis proof suggestion
A live Telegram transcript would materially confirm that plain assistant text is delivered in a group chat when the model skips the message tool. A maintainer can ask Mantis to capture proof by posting a new PR comment that starts with the OpenClaw Mantis account mention, followed by:

telegram live: verify that a Telegram group reply in message_tool_only mode still delivers assistant text when the model does not call the message tool.

Risk before merge
Why this matters: - The patch changes delivery suppression bypass behavior in message_tool_only chats; if a delivery path is missing from hasSuccessfulSideEffectDelivery, users could see duplicate final text.

  • The PR body lists live proof and local test runs, but the diff does not add a focused regression test for this new fallback branch.

Maintainer options:

  1. Accept The Scoped Fallback (recommended)
    Maintainers can merge if they are comfortable that the existing side-effect delivery helper covers the real duplicate-delivery cases.
  2. Add Focused Coverage First
    Add a narrow agent-runner regression test for message_tool_only assistant text with no side-effect delivery, plus a negative case where a side effect exists.

Next step before merge
This active PR has no discrete blocking review finding; the remaining action is maintainer merge judgment, not a repair-lane task.

Security
Cleared: The diff only changes in-process reply payload metadata and does not touch secrets, dependencies, CI, permissions, or external code execution.

Review details

Best possible solution:

Land the scoped fallback with maintainer acceptance of the live Telegram proof, ideally with a focused regression test proving the fallback marks only undelivered assistant text.

Do we have a high-confidence way to reproduce the issue?

Yes from source inspection and PR evidence, but I did not run a live Telegram repro. Current main suppresses unmarked final replies in message_tool_only mode, and the PR body provides live before/after Telegram trajectory data.

Is this the best way to solve the issue?

Yes, likely. The patch stays in the agent-runner delivery boundary and reuses the existing side-effect helper; focused regression coverage would make it stronger.

Label justifications:

  • P2: This fixes a limited but user-visible Telegram message-loss path.
  • merge-risk: 🚨 message-delivery: The diff changes whether final assistant text is delivered or suppressed in message-tool-only chats.

What I checked:

Likely related people:

  • Ayaan Zaidi: Blame shows the current agent-runner, dispatch-from-config, and Telegram message-tool-only wiring in the reviewed checkout came from the recent Telegram queue blocker commit. (role: recent area contributor; confidence: high; commits: d75e16a1b920; files: src/auto-reply/reply/agent-runner.ts, src/auto-reply/reply/dispatch-from-config.ts, extensions/telegram/src/bot-message-dispatch.ts)
  • Patrick Erichsen: Recent merged Telegram progress-preview work touched the Telegram dispatch flow and tests adjacent to this delivery behavior. (role: adjacent owner; confidence: medium; commits: d60ab485114a; files: extensions/telegram/src/bot-message-dispatch.ts, extensions/telegram/src/bot-message-dispatch.test.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 3bc728eaa993.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels May 19, 2026
@clawsweeper clawsweeper Bot mentioned this pull request May 19, 2026
Closed
@pashpashpash

pashpashpash commented May 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Instead of adding fallbacks, we should fix the actual root cause and understand why the model doesn't use the message tool as instructed.

This was referenced May 20, 2026
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Telegram group chat: model responses silently dropped when message tool not called

2 participants

@billehunt @pashpashpash