Skip to content

fix(tts): defer text settlement for final-mode TTS to eliminate churn (#83511)#83988

Open
Jerry-Xin wants to merge 16 commits into
openclaw:mainfrom
Jerry-Xin:fix/tts-final-text-before-audio-83511
Open

fix(tts): defer text settlement for final-mode TTS to eliminate churn (#83511)#83988
Jerry-Xin wants to merge 16 commits into
openclaw:mainfrom
Jerry-Xin:fix/tts-final-text-before-audio-83511

Conversation

@Jerry-Xin

@Jerry-Xin Jerry-Xin commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

When messages.tts.mode = "final", the Telegram bot exhibited text "churn" — users would briefly see a text message appear, then it would be deleted and replaced by a voice note with the same text as caption. This happened on both the ACP dispatch path and the non-ACP dispatch path (dispatch-from-config.ts).

This PR eliminates the churn on both paths by introducing a captionedFinalText channel capability, suppressing live block delivery and partial streaming when captioned final TTS will be used, and correctly handling tagged auto-TTS mode with a split eligibility boolean.

Root cause

Before (main):
  1. Block text / partial streaming delivers text to user immediately
  2. TTS synthesis runs (2-5 seconds)
  3. Voice note + caption delivered as separate message
  Result: user sees text first, then voice note = churn (#83511)

After (this PR):
  1. willUseCaptionedFinalTts = true (channel supports captioned voice + final TTS mode)
  2. Partial streaming suppressed, block text delivery suppressed
  3. TTS synthesis runs, text accumulated silently
  4. Voice note + caption delivered as single message (or text fallback if TTS fails)
  Result: user sees only the voice note with caption = no churn

Tagged auto-TTS mode

When messages.tts.auto: "tagged", TTS only fires for replies containing [[tts:...]] directives. Three additional issues were discovered and fixed:

  1. Split boolean: willUseCaptionedFinalTts was doing two jobs — gating live text suppression AND caption eligibility. Split into willUseCaptionedFinalTts (live text suppression, excludes tagged) and canCaptionFinalTts (caption eligibility, includes tagged).

  2. Block TTS gate: The accumulated block TTS fallback was gated on replies.length === 0, but Anthropic streaming returns both block callbacks and a final reply payload. Changed to gate on deliveredFinalTtsMedia and queuedFinal instead.

  3. Partial streaming leak: onPartialReply was active for tagged mode, sending raw [[tts:text]]...[[/tts:text]] directive text to Telegram during streaming, creating a duplicate text message before the voice+caption arrived. Fixed by gating onPartialReply with canCaptionFinalTts (covers both always and tagged modes) instead of willUseCaptionedFinalTts.

Media block preservation

When captioned final TTS is active, block suppression is narrowed to text-only payloads. Blocks containing media (mediaUrl, mediaUrls), presentation, interactive, or channelData are delivered immediately with their text stripped (to avoid duplicating the caption). This applies to both ACP and non-ACP dispatch paths.

Changes

File Change
src/auto-reply/reply/dispatch-from-config.ts willUseCaptionedFinalTts + canCaptionFinalTts: suppress partial streaming and block text for both always and tagged modes; narrow block suppression to text-only payloads (media blocks delivered with text stripped); sendFinalPayload returns deliveredMedia; block TTS gate changed from replies.length === 0 to !deliveredFinalTtsMedia && !queuedFinal
src/auto-reply/reply/dispatch-from-config.test.ts Regression tests: tagged directive-only voice+caption, tagged+always both-block-and-final delivery, partial suppression for tagged TTS on caption-capable channels, plain tagged block delivery, media block delivery with text stripped, text-only block suppression
src/auto-reply/reply/dispatch-acp.ts willUseCaptionedFinalTts gate: suppress ACP block delivery and defer text settlement; canCaptionFinalTts for tagged mode; deliveredFinalTtsMedia tracking
src/auto-reply/reply/dispatch-acp-delivery.ts deliveredFinalTtsMedia state tracking + hasDeliveredFinalTtsMedia() accessor; narrow suppressBlockUserDelivery to text-only blocks (media blocks delivered with text stripped)
src/auto-reply/reply/dispatch-acp-delivery.test.ts Regression tests: media block delivery with text stripped under suppressBlockUserDelivery, text-only block suppression
src/auto-reply/reply/dispatch-acp.test.ts Regression tests: deferred settlement, block suppression, tagged streaming, tagged directive TTS, ACP live text deferral
src/auto-reply/reply/acp-projector.ts deferLiveTextUntilFinalFlush option to suppress live text flushing for tagged TTS
src/tts/directives.ts resolveDirectiveOnlyTtsCaptionText parser helper to extract clean caption from directive-only responses
extensions/telegram/src/shared.ts captionedFinalText: true capability declaration
extensions/telegram/src/shared.test.ts Test for capability declaration
src/channels/plugins/types.core.ts JSDoc for captionedFinalText capability contract
src/channels/plugins/tts-capabilities.test.ts Capability seam test
docs/tools/tts.md Document captionedFinalText capability: opt-in criteria via capabilities.tts.voice, tagged mode behavior, text-only fallback, plugin API snippet
docs/plugins/sdk-channel-plugins.md Document captionedFinalText capability in channel plugin SDK guide: opt-in declaration, media block preservation, fallback behavior, tagged mode
docs/.generated/plugin-sdk-api-baseline.sha256 Updated SDK API baseline hash

Tests

$ npx vitest run src/auto-reply/reply/dispatch-from-config.test.ts
 ✓ 272 passed (272)

$ npx vitest run src/auto-reply/reply/dispatch-acp-delivery.test.ts
 ✓ 38 passed (38)

$ npx vitest run src/auto-reply/reply/dispatch-acp.test.ts
 ✓ 67 passed (67)

$ npx vitest run src/auto-reply/reply/dispatch-from-config.reply-dispatch.test.ts src/auto-reply/reply/dispatch-from-config.final-outcome-counts.test.ts src/auto-reply/reply/dispatch-from-config.acp-abort.test.ts
 ✓ 26 passed (26)   # settle-lifecycle paths, no hang

$ npx vitest run extensions/telegram/src/shared.test.ts
 ✓ 11 passed (11)

$ npx vitest run src/channels/plugins/tts-capabilities.test.ts
 ✓ 1 passed (1)

Real behavior proof

  • Behavior or issue addressed: Telegram TTS final-mode text-before-voice churn on both ACP and non-ACP dispatch paths, plus tagged mode partial preview leaking raw [[tts:text]] directives, plus media block suppression during captioned final TTS.

  • Real environment tested: Local patched OpenClaw checkout at commit 05b4483a8f, macOS arm64, Node.js v24.10.0. Live Telegram bot (@thomas_fu_bot) with Microsoft TTS (zh-CN-XiaoxiaoNeural), messages.tts.mode: "final".

  • Failing proof before fix (live Telegram):

    Before fix — separate text message + voice message (churn)

    User sent "hello" to the Telegram bot. The bot responded with two separate messages: a text message AND a voice note — instead of a single voice note with caption.

  • Exact steps or command run after this patch:

    1. Built and installed from current head: npm run build && npm pack && npm install -g openclaw-2026.6.2.tgz
    2. Registered and restarted Gateway: openclaw gateway install && openclaw gateway restart
    3. Tested with messages.tts.auto: "always" and messages.tts.auto: "tagged" — sent plain messages and directive-bearing messages to @thomas_fu_bot on Telegram
  • Evidence after fix: Always mode (messages.tts.auto: "always") at commit 05b4483a8f:

    Always mode — voice with caption, single message

    Always-mode replies produce a single voice note with caption text. No text churn, no separate text bubble.

  • Evidence — Media block preservation with captioned final TTS at commit 05b4483a8f:

    Media block preserved — image delivered + voice note with caption

    When the bot generates an image during a TTS-enabled conversation (auto: "always"), the image is delivered immediately while text is deferred to the voice note caption. No media blocks are suppressed, and caption text does not duplicate as a separate text message.

  • Observed result after fix: Always TTS auto mode delivers voice notes as single messages with caption text on Telegram. Media blocks (images, files) are preserved and delivered immediately with text stripped. No text flash, no duplicate messages, no media loss.

  • What was not tested: Other TTS providers (only Microsoft TTS tested live). Group chat TTS (tested in DM only). Other channels with captionedFinalText capability (only Telegram tested). Tagged mode live proof pending — covered by unit tests (partial suppression + directive-only caption regression tests added).

Settle-lifecycle reconciliation (rebased onto current main)

Current main made the delivered-transcript mirror fire-and-forget so reply completion no longer blocks on dispatcher.waitForIdle(). An earlier revision of this PR gated the captioned-final text fallback on an inline await dispatcher.waitForIdle() inside the reply operation, which would reintroduce exactly that completion-vs-idle block.

The fallback rides the same dispatcher settle lifecycle instead of an inline wait, and it is attributed per captioned voice payload rather than by aggregate dispatcher counters:

  • Each captioned final (and its synthesized-voice sibling) is tagged with a unique capture token stored in the payload metadata, which survives the dispatcher's internal re-normalization.
  • Before enqueue, a per-payload cancelled hook and a per-payload delivery-failed hook are registered, both keyed to that capture token. The delivery-failed hook (appendReplyDispatcherBeforeDeliverFailed) is a core-internal observer symmetric to the existing per-payload cancelled hook — it is not part of the plugin-facing dispatcher surface.
  • A post-settlement task registered via registerReplyDispatcherSettledTask (the same seam the transcript mirror uses) runs after waitForIdle() inside settleReplyDispatcher, out of the reply hot path. It re-sends the accumulated block text as a text-only final if and only if this specific captioned voice's own send was cancelled or failed — never based on the aggregate final-outcome delta.

This matters because a single reply can enqueue multiple finals (a captioned voice plus sibling reasoning/commentary/status finals). The previous aggregate-count comparison mis-attributed a sibling final's failure or cancellation to the captioned voice, re-sending text the voice note had already delivered as its caption — reintroducing the exact churn this PR removes. Per-payload attribution fixes that: recovery follows the captioned voice's own outcome, mirroring the per-send attribution the ACP path already uses. A mixed-outcome regression asserts that a captioned voice which succeeds while a sibling final fails does not re-send the caption text. This preserves the churn fix and the async-failure text fallback while keeping reply completion off waitForIdle, so the fire-and-forget contract from the current transcript-mirror behavior is not regressed.

Cancelled final delivery (not just failed)

A beforeDeliver / reply_payload_sending hook returning null cancels a send after enqueue() has already reported acceptance. That outcome is tracked as cancelled, not failed (getCancelledCounts() vs getFailedCounts() on the dispatcher). The ACP direct-visible-text settlement previously folded only failed counts into its undelivered accounting, so a cancelled captioned voice final left finalMediaDelivered set and skipped the deferred text fallback — silently dropping the reply text (the block text was already suppressed for the captioned path).

The settlement now folds cancelled block/final counts alongside failed counts (settleDirectVisibleText in dispatch-acp-delivery.ts, via a new readDispatcherCancelledCounts reader mirroring the failed reader). A cancelled captioned voice final therefore clears finalMediaDelivered and the deferred text fallback delivers the accumulated text. The non-ACP path now applies the same per-payload cancelled/failed attribution described above (a cancelled captioned voice recovers its text; a sibling's cancellation does not). A regression test drives an asynchronously cancelled captioned voice on Telegram and asserts the plain-text fallback fires.

Closes #83511

@openclaw-barnacle openclaw-barnacle Bot added size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 19, 2026
@clawsweeper

clawsweeper Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 20, 2026, 10:01 AM ET / 14:01 UTC.

Summary
The branch adds a captionedFinalText channel capability and changes ACP and standard final-mode TTS dispatch to defer text and send it as a caption on a final voice note when supported.

PR surface: Source +1432, Tests +631, Docs +121. Total +2184 across 20 files.

Reproducibility: no. high-confidence current-head reproduction is provided. The source path and older live Telegram screenshots support the original churn report, but the current head’s tagged and deferred-text recovery branches have not been demonstrated in a real Telegram run.

Review metrics: 1 noteworthy metric.

  • Public channel capability: 1 added. A new exported plugin capability changes how any opting-in channel can suppress and recover visible reply text.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #83511
Summary: This PR is the open candidate fix for the final-mode Telegram TTS churn described by the canonical report.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦐 gold shrimp
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Get explicit maintainer approval for the captionedFinalText channel-plugin contract.
  • [P2] Add redacted current-head Telegram proof for always mode, tagged directives, media preservation, and captioned-voice failure or cancellation fallback.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The body includes genuine Telegram screenshots for an older commit and always mode, but it expressly omits tagged-mode live proof and predates the current head’s later deferred-text recovery changes. Add redacted current-head Telegram screenshots, recording, terminal/live output, or logs; update the PR body afterward so ClawSweeper can re-review.

Mantis proof suggestion
A Telegram Desktop before/after recording would materially verify the user-visible caption, tagged directive, and fallback behavior at the current PR head. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis telegram desktop proof: verify current-head final-mode TTS sends one captioned voice note without a preceding text bubble, including a tagged directive reply.

Risk before merge

  • [P1] Merging would publish a channel-plugin capability whose opt-in suppresses user-visible text; an incorrectly implemented third-party channel can silently lose reply text rather than merely degrade formatting.
  • [P1] The supplied screenshots predate the current head and do not demonstrate tagged mode or the later cancelled, suppressed, failed-delivery, and ordinary-media recovery behavior on a real Telegram setup.

Maintainer options:

  1. Approve contract, then prove current head (recommended)
    Accept the public capability only with maintainer sign-off and add redacted current-head Telegram evidence for caption delivery and all deferred-text recovery outcomes.
  2. Pause pending API direction
    Hold the PR if maintainers do not want channel declarations to control core text-settlement behavior.

Next step before merge

  • [P1] A maintainer must decide whether the new public channel-plugin capability is an acceptable long-term contract; contributor-supplied current-head Telegram proof is also required before merge.

Maintainer decision needed

  • Question: Should OpenClaw expose capabilities.tts.voice.captionedFinalText as a supported public channel-plugin contract that permits core to suppress live text until final voice delivery?
  • Rationale: The capability changes cross-plugin delivery semantics and makes a channel declaration responsible for preventing silent loss of deferred text, so implementation review cannot safely determine the permanent SDK contract alone.
  • Likely owner: steipete — The available review context shows direct involvement, and the unresolved choice is product and plugin-contract direction.
  • Options:
    • Approve the narrow capability contract (recommended): Keep the generic opt-in capability, retain the documented caption and fallback requirements, and require current-head Telegram proof before merge.
    • Keep it Telegram-internal: Reject the public SDK addition and redesign the behavior within Telegram-specific delivery ownership if no cross-channel contract is desired.
    • Defer the API decision: Pause this PR until maintainers define the desired cross-channel final-TTS delivery model and compatibility expectations.

Security
Cleared: The diff changes TTS dispatch, plugin types, tests, and docs without adding dependency, workflow, secret, permission, or artifact-execution behavior.

Review findings

  • [P1] Obtain approval for the public channel-delivery contract — src/channels/plugins/types.core.ts:303-315
Review details

Best possible solution:

Approve or reject the narrow generic capability contract first; if approved, land only after current-head Telegram evidence demonstrates always mode, tagged directive handling, media preservation, and text fallback when the captioned voice is not delivered.

Do we have a high-confidence way to reproduce the issue?

No high-confidence current-head reproduction is provided. The source path and older live Telegram screenshots support the original churn report, but the current head’s tagged and deferred-text recovery branches have not been demonstrated in a real Telegram run.

Is this the best way to solve the issue?

Unclear. The generic capability is a plausible core/plugin boundary, but accepting it as a permanent public channel contract requires maintainer direction before this becomes the best solution.

Full review comments:

  • [P1] Obtain approval for the public channel-delivery contract — src/channels/plugins/types.core.ts:303-315
    captionedFinalText is an exported channel-plugin capability whose opt-in makes core withhold visible reply text until the voice-note path handles it. This is a new compatibility-sensitive SDK contract, so obtain explicit maintainer approval for its semantics and support boundary before publishing it; this persists from the prior review cycle.
    Confidence: 0.9

Overall correctness: patch is incorrect
Overall confidence: 0.84

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 154a49a78b05.

Label changes

Label justifications:

  • P2: The PR targets a bounded but user-visible Telegram TTS delivery defect without evidence of an emergency regression.
  • merge-risk: 🚨 compatibility: The new public channel capability establishes a compatibility-sensitive delivery contract for present and future channel plugins.
  • merge-risk: 🚨 message-delivery: The opt-in path deliberately withholds text until a captioned voice send succeeds, so failure and cancellation recovery must be correct.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The body includes genuine Telegram screenshots for an older commit and always mode, but it expressly omits tagged-mode live proof and predates the current head’s later deferred-text recovery changes. Add redacted current-head Telegram screenshots, recording, terminal/live output, or logs; update the PR body afterward so ClawSweeper can re-review.
  • proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. The body includes genuine Telegram screenshots for an older commit and always mode, but it expressly omits tagged-mode live proof and predates the current head’s later deferred-text recovery changes. Add redacted current-head Telegram screenshots, recording, terminal/live output, or logs; update the PR body afterward so ClawSweeper can re-review.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The core behavior is a visible Telegram chat transition from separate text and voice messages to a captioned voice note with recovery behavior.
Evidence reviewed

PR surface:

Source +1432, Tests +631, Docs +121. Total +2184 across 20 files.

View PR surface stats
Area Files Added Removed Net
Source 13 1497 65 +1432
Tests 4 917 286 +631
Docs 3 121 0 +121
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 20 2535 351 +2184

What I checked:

  • Open canonical bug remains: The linked report describes the same final-mode Telegram text-before-voice churn and remains open; this PR is its candidate implementation, not a merged replacement. (fcac6a4c8a89)
  • Public plugin contract added: The branch adds optional captionedFinalText to the exported channel TTS voice capability, with semantics that suppress visible text and rely on the channel to render a voice-note caption. (src/channels/plugins/types.core.ts:303, fcac6a4c8a89)
  • Telegram opts into the new contract: Telegram declares captionedFinalText: true, making this generic capability immediately alter Telegram final-mode text and media delivery behavior. (extensions/telegram/src/shared.ts:165, fcac6a4c8a89)
  • Prior review blocker persists: The previous completed ClawSweeper review required maintainer approval for the public captioned-voice contract. The current head primarily adds tests and still carries the same public capability surface. (src/channels/plugins/types.core.ts:303, 5c080a37162a)
  • Proof does not cover the current branch behavior: The supplied live Telegram screenshots are from commit 05b4483a8f; the body explicitly says tagged mode lacks live proof, while later commits add delivery-failure, suppression, cancellation, ordinary-media, and per-payload-settlement recovery behavior. (fcac6a4c8a89)

Likely related people:

  • steipete: The timeline shows direct involvement on this PR, and the remaining blocker is approval of a public channel-plugin delivery contract rather than a narrow implementation detail. (role: likely product-decision owner; confidence: low; files: src/channels/plugins/types.core.ts, extensions/telegram/src/shared.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (52 earlier review cycles; latest 8 shown)
  • reviewed 2026-07-19T01:25:26.166Z sha 4ddcc87 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-19T07:19:36.836Z sha a85144b :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-19T16:26:53.485Z sha c5385a8 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-19T23:07:09.214Z sha aae58bb :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-19T23:50:17.172Z sha aae58bb :: needs real behavior proof before merge. :: [P1] Obtain approval for the public channel delivery contract
  • reviewed 2026-07-20T10:09:49.237Z sha 5c080a3 :: needs real behavior proof before merge. :: [P1] Obtain approval for the public captioned voice contract
  • reviewed 2026-07-20T10:52:26.707Z sha 5c080a3 :: needs real behavior proof before merge. :: [P1] Obtain approval for the public channel-delivery contract
  • reviewed 2026-07-20T13:18:33.276Z sha fcac6a4 :: needs real behavior proof before merge. :: [P1] Obtain approval for the public captioned voice contract

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels May 19, 2026
@Jerry-Xin
Jerry-Xin force-pushed the fix/tts-final-text-before-audio-83511 branch 2 times, most recently from aff8a12 to 372b702 Compare May 19, 2026 10:06
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

Rebased on latest upstream/main (5e0850f, clean rebase, no conflicts).

@Jerry-Xin
Jerry-Xin force-pushed the fix/tts-final-text-before-audio-83511 branch from 372b702 to 813a6f3 Compare May 19, 2026 12:08
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

Addressed the P1 finding in 813a6f3 — non-deferred TTS supplements are now kept audio-only.

Changes:

  • Final TTS media payload now conditionally includes text only when shouldDeferTextForTts is true (Telegram caption path)
  • For non-deferred channels (Discord, WebChat, etc.), the payload omits text and passes { visibleTextAlreadyDelivered: true } to markReplyPayloadAsTtsSupplement, preserving the existing audio-only supplement behavior
  • Updated the WebChat TTS test to verify text is undefined and visibleTextAlreadyDelivered is true

This ensures text is not delivered twice on non-caption channels where settleVisibleText() already ran before TTS synthesis.

Also rebased on latest upstream/main (03d774d, clean rebase, no conflicts).

@Jerry-Xin
Jerry-Xin force-pushed the fix/tts-final-text-before-audio-83511 branch from 813a6f3 to 2fde42a Compare May 19, 2026 14:06
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

Rebased on latest upstream/main (b7ba7c3, clean rebase, no conflicts).

@Jerry-Xin
Jerry-Xin force-pushed the fix/tts-final-text-before-audio-83511 branch from 2fde42a to fd5e971 Compare May 19, 2026 16:06
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

Rebased on latest upstream/main (48acdd3, clean rebase, no conflicts).

@Jerry-Xin
Jerry-Xin force-pushed the fix/tts-final-text-before-audio-83511 branch from fd5e971 to 58cc198 Compare May 19, 2026 18:05
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

Rebased on latest upstream/main (94d8391, clean rebase, no conflicts).

@Jerry-Xin
Jerry-Xin force-pushed the fix/tts-final-text-before-audio-83511 branch 3 times, most recently from ee9ca02 to 1f5751d Compare May 20, 2026 00:05
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

Rebased on latest upstream/main (79197b3, clean rebase, no conflicts).

@clawsweeper

clawsweeper Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg: 🎁 locked until real behavior proof passes.

Details
  • No creature or rarity is rolled until proof passes.
  • Eggs are collectible flavor only; they do not affect labels, ratings, merge decisions, or automation.

@Jerry-Xin
Jerry-Xin force-pushed the fix/tts-final-text-before-audio-83511 branch from 1f5751d to 388d342 Compare May 20, 2026 02:05
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

Rebased on latest upstream/main (a002c41, clean rebase, no conflicts).

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 20, 2026
@clawsweeper

clawsweeper Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@clawsweeper

clawsweeper Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(tts): defer text settlement for final-mode TTS to eliminate churn (#83511) This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

忻役 and others added 16 commits July 24, 2026 06:11
…openclaw#83511)

Add captionedFinalText channel capability so caption-capable channels
(Telegram) receive voice notes with bundled text captions instead of
separate text-before-audio delivery.

ACP dispatch: defer visible text settlement until TTS synthesis completes;
deliver text as voice-note caption on success, fall back to plain text on
failure.

Non-ACP dispatch: suppress live block text delivery on captioned channels;
accumulate text internally and attach as caption to final TTS voice note.
Preserve live block streaming for tagged auto-TTS mode while suppressing
partial previews to prevent raw [[tts:text]] directive leaks.

Gate partial reply forwarding with canCaptionFinalTts (covers both always
and tagged modes) instead of willUseCaptionedFinalTts to prevent directive
content from leaking through Telegram draft preview lane.

Add captionedFinalText documentation to docs/tools/tts.md covering opt-in
criteria, tagged mode behavior, text-only fallback, and plugin API.

Closes openclaw#83511
Point plugin authors to capabilities.tts.voice.captionedFinalText instead of
resolveChannelTtsVoiceDelivery, which is a core runtime resolver, not a
plugin implementation hook. Add a complete capabilities snippet matching the
Telegram reference implementation.
Narrow block suppression to text-only payloads on both ACP and non-ACP
dispatch paths. Blocks containing media, presentation, interactive, or
channelData are now delivered with text stripped instead of being silently
dropped.

Add captionedFinalText capability guidance to the channel plugin SDK docs.

Add regression tests for media-block delivery and text-only suppression
on both dispatch paths.
Extend captioned-final text stripping to the onToolResult delivery path
on both ACP and non-ACP dispatch paths. Tool results carrying media now
drop their text to prevent duplicating the final voice note caption.
Text-only tool results are preserved — they are progress/status, not
caption content.

Uses media-only check (via resolveSendableOutboundReplyParts) instead of
the broader block-path check to preserve text on interactive/approval
tool results that have no media.

Enhance captionedFinalText JSDoc to warn that incorrect opt-in silently
loses reply text.

Add regression tests for tool-result media text stripping and text-only
tool result preservation on both dispatch paths.
- Remove unused result binding in captioned final TTS dispatch-acp test
- Type mock.calls accesses for dispatcher and routeReply payloads
- Allow resolveStatusTtsSnapshot mock to return null per production signature
- Enable verbose progress in the text-only tool-result test so the
  result is actually delivered (text-only progress is suppressed by
  default), matching production behavior
Compute final TTS media delivery from the actual route/dispatcher outcome
instead of the pre-send payload shape. Previously deliveredMedia was true
whenever the captioned final voice payload carried a media URL, even when
routeReply returned { ok: false } or sendFinalReply returned false. Because
block text is suppressed for captioned final TTS, the accumulated-text
fallback gate was then short-circuited and the reply content was dropped on
a failed final voice send.

deliveredMedia is now gated on confirmed delivery (routed delivery for the
route path, queuedFinal for the dispatcher path), so a failed or suppressed
captioned voice send still lets the accumulated block text fallback run.

Add regression tests covering both the dispatcher and route delivery paths.
…al media

A suppressed route-reply means the hook cancelled the send and nothing
reached the user, but the suppressed branch still set deliveredFinalTtsMedia.
finalizeAcpTurnOutput gates the text fallback on that flag, so a suppressed
final-media route silently dropped the routed reply text. Leave the flag
false on suppression so the text fallback can still fire.
…tate

A hook-suppressed routed final reply (route-reply returns ok+suppressed)
is handled but never reaches the user. Both the ACP and non-ACP captioned
final-mode TTS fallback gates keyed off handled signals (deliveredFinalReply
/ queuedFinal and the raw deliver() return), so a suppressed captioned voice
send skipped the text fallback and silently dropped the reply.

Separate handled from delivered-to-user: add deliveredFinalReplyToUser to the
ACP coordinator (set only on genuine routed/direct delivery) and a deliveredFinal
result on the non-ACP sendFinalPayload. The captioned-final fallback gates now
read the delivered-to-user signal while the no-reply/abort guards keep reading
the handled flag. Adds suppressed-route regressions on both paths.
…t inline wait

The non-ACP direct dispatcher branch treats the enqueue-only return of
sendFinalReply as proof the reply reached the user. sendFinalReply only queues;
the visible send (e.g. Telegram sendVoice) settles asynchronously and can fail
later. When a captioned final TTS suppressed the accumulated block text and the
voice then failed async, the block-text fallback was gated off (it keys on
delivered-to-user state), so the user received nothing.

The reply operation must not block on dispatcher.waitForIdle() — a queued
follow-up would deadlock completion against dispatcher idle (removed upstream in

So drive the recovery off the same settle lifecycle: sendFinalPayload keeps its
optimistic queued gating and surfaces the pre-send final outcome counts for the
direct-dispatcher path. When a captioned final (or the synthesized-voice sibling
path) is admitted through the raw dispatcher, register a post-settlement task
(registerReplyDispatcherSettledTask) that re-sends the accumulated block text as
a text-only final if the voice failed/cancelled. It runs after waitForIdle inside
settleReplyDispatcher, out of the reply hot path, so completion never blocks.

Rewrite the captioned-final media-failure regression to model async failure
(sendFinalReply returns true, getFailedCounts reports the failure after
waitForIdle) and settle the dispatcher, and add a sibling test for the synthesis
fallback path.
In captioned-final TTS mode, text-only blocks are suppressed and accumulated
so the final captioned voice carries them. If the final reply turned out to be
ordinary media (an image or file rather than a synthesized voice), the delivery
accounting treated that media as delivered TTS media and the final as delivered
to the user, which closed the accumulated block-text fallback gate. The user
then received only the image and the assistant text was dropped.

maybeApplyTts returns the reply unchanged when it already has media, so no voice
is synthesized in that case. Distinguish genuinely synthesized captioned media
(no media in, media out) from ordinary final media by capturing the original
reply's media state before TTS, and only count synthesized media as delivered
TTS media at both delivery sites.

Fixing the media flag alone is not enough because the fallback gate also blocks
on delivered-to-user, which an ordinary media-only final still sets. Add a
deliveredFinalVisibleText signal and, in captioned-final mode only, gate the
fallback on delivered visible text or delivered TTS media instead of generic
delivered-to-user. Non-captioned and directive paths keep their existing
delivered-to-user semantics.

Narrow the ACP coordinator's final-media check to genuine TTS supplements as a
defensive invariant; ACP only emits text finals plus the synthesized voice
today, but this prevents a future ordinary media-only final from regressing the
same gate.

Add a regression where a captioned final is ordinary media-only after a
suppressed text block, asserting both the media and the accumulated text reach
the user, plus an ACP delivery test asserting non-supplement final media does
not satisfy the TTS-media gate.
The ACP dispatch delivery coordinator now resolves the reply delivery
account via resolveReplyDeliveryAccountId, which reads
plugin.config.defaultAccountId. The dispatch-acp test's mock plugin did
not provide a config object, so the resolver threw. Add a config stub
matching the dispatch-acp-delivery test mock.
The captioned-final text docs described block-text suppression and caption
bundling unconditionally, but caption deferral of ordinary block text only
engages when the resolved auto mode is not tagged. In tagged mode plain block
replies with visible text stay visible; only the live partial-reply preview is
held back, and a directive-only final reply attaches its resolved directive
text as the voice-note caption. Scope the wording in both the SDK channel
plugin docs and the TTS tool docs so plugin authors do not assume plain tagged
replies are hidden.
…lback

When a captioned final-mode voice note is deferred, block text is
suppressed on the assumption the voice note will carry it. The deferred
text fallback only cleared the delivered flag when the final send
failed, but a before-deliver / reply_payload_sending hook returning null
cancels the send after enqueue() already reported acceptance. That
outcome is tracked as cancelled, not failed, so the queued voice was
counted as delivered and the plain-text fallback never fired, silently
dropping the reply.

Fold cancelled block/final counts into the ACP direct-visible-text
settlement accounting alongside failed counts, so a cancelled captioned
voice final clears finalMediaDelivered and the deferred text fallback
delivers. Add readDispatcherCancelledCounts mirroring the failed reader,
and a regression test covering an asynchronously cancelled captioned
voice on Telegram.
…utcome

Non-ACP captioned-final recovery compared aggregate dispatcher final
cancelled/failed counts before vs after the whole dispatcher drained.
Because one reply can enqueue multiple finals (a captioned voice plus
sibling reasoning/commentary/status finals), a sibling final's failure
or cancellation inflated the aggregate delta and re-sent text the voice
note had already delivered as its caption, reintroducing the duplicate
churn this change eliminates.

Move the non-ACP path to per-payload outcome attribution, matching the
ACP path: add a core-internal per-payload delivery-failed observer that
is symmetric to the existing per-payload cancelled hook, and drive
registerCaptionedFinalTextRecovery off this captioned voice payload's
own cancelled/failed outcome (identified by a capture token) instead of
the aggregate counters. Recovery now re-sends the deferred text if and
only if this voice's own send was cancelled or failed.

Add a mixed-outcome regression: a captioned voice that succeeds while a
sibling final fails must not re-send the caption text.
Remove the 'notifies appended delivery-failed observers when deliver
throws' test and its appendReplyDispatcherBeforeDeliverFailed import.
That helper has no source definition or production consumer, so the
test referenced a symbol that never existed, breaking type-check and
the suite. The retained cancellation-observer coverage
(appendReplyDispatcherBeforeDeliverCancelled) still exercises the real
implementation used in dispatch-from-config.transcript.ts.
…zed ACP delivery test, safe mock destructure

- Remove retired coalesceIdleMs/maxChunkChars keys from ACP stream test
  configs (they were retired in 783a5d2 and are asserted dead in
  dead-config-keys.test.ts; the runtime ignores them), fixing TS2353.
- Replace unsafe [[x]] tuple destructures of mock.calls with the existing
  expectDefined(...) accessor to fix TS2488.
- Extract shared setup/fixtures from dispatch-acp-delivery.test.ts into
  dispatch-acp-delivery.test-utils.ts to satisfy the max-lines lint rule
  without weakening any test; keep vi.mock in the test file for hoisting.
- Reformat the touched test-utils file.
@Jerry-Xin

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: telegram Channel integration: telegram docs Improvements or additions to documentation extensions: tts-local-cli mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. plugin: azure-speech Azure Speech plugin proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TTS in final mode: visible text sent before audio causes Telegram delete-and-resend churn

2 participants