Skip to content

fix: preserve steered audio for inbound TTS#95596

Merged
steipete merged 1 commit into
mainfrom
fix/pr-94049-steered-inbound-audio
Jul 7, 2026
Merged

fix: preserve steered audio for inbound TTS#95596
steipete merged 1 commit into
mainfrom
fix/pr-94049-steered-inbound-audio

Conversation

@mcaxtr

@mcaxtr mcaxtr commented Jun 21, 2026 •
edited by steipete
Loading

Copy link
Copy Markdown
Member

What Problem This Solves

When tts.auto is set to "inbound", OpenClaw should use voice output for replies that answer inbound audio. A text-started reply run could later accept steered inbound audio while the active run was still streaming, but TTS delivery kept the original text message's audio classification. The agent could therefore answer the later audio with a text-only reply.

Fixes #76831.

Why This Change Was Made

The accepted-audio fact belongs to the current reply operation, not durable session state. This rewrite latches that fact on ReplyOperation only after the active run accepts the steer. Automatic final, block, and tool delivery reads it at TTS time; embedded message tools receive a dynamic getter so message_tool_only delivery reads the same current operation state.

That leaves one canonical owner, avoids persisted session compatibility, removes the original embedded-queue option threading, and keeps the behavior channel-agnostic.

User Impact

Users with tts.auto: "inbound" receive voice output when a later inbound audio message is accepted into an active text-originated run, including configurations that deliver replies through the message tool.

Rejected steers, later text messages, and unrelated reply operations do not change delivery modality.

Evidence

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S maintainer Maintainer-authored PR labels Jun 21, 2026
@clawsweeper

clawsweeper Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 6, 2026, 11:31 PM ET / 03:31 UTC.

Summary
This PR threads a steered inbound-audio marker through active embedded-run queueing, latches it on the in-memory reply operation after acceptance, and uses it when resolving final inbound-auto TTS.

PR surface: Source +28, Tests +109. Total +137 across 12 files.

Reproducibility: yes. source inspection gives a high-confidence reproduction path: a text-originated active run can accept an audio follow-up, but current main does not carry that audio fact into final TTS dispatch. I did not run a live WhatsApp timing repro in this read-only review.

Review metrics: 2 noteworthy metrics.

  • Config/default surfaces: 0 added, 0 changed, 0 removed. A reviewer asked whether config behavior changes; the diff changes runtime metadata and tests, not config, defaults, or migrations.
  • Operation-local TTS markers: 1 added. The new reply-operation marker controls final inbound-auto TTS modality, so its lifecycle is the key maintainer review point.

Stored data model
Persistent data-model change detected: vector/embedding metadata: src/auto-reply/reply/agent-runner.media-paths.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #76831
Summary: This PR is the current candidate fix for the canonical WhatsApp active-run inbound-audio TTS issue; the earlier persisted-session PR was closed as superseded, and the merged message-tool PR covers only adjacent TTS metadata paths.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🦐 gold shrimp
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Rebase and adapt the patch to current main’s ReplyBackendQueueMessageOptions shape.
  • Rerun focused reply-run, embedded-runner, dispatch, tsgo, and test-type checks on the merge head.
  • [P2] Add redacted WhatsApp active-steer audio-to-voice proof if maintainers want transport-level confidence.

Risk before merge

  • [P1] The branch is currently conflicting against current main, which has added broader queue-message options since this PR was written.
  • [P1] The change intentionally affects final reply modality for accepted active steer-mode turns, so a lifecycle mistake could send voice when text was expected or continue suppressing voice when audio should be used.
  • [P1] The PR body has focused command proof, but no live WhatsApp active-steer audio-to-voice roundtrip on this head.

Maintainer options:

  1. Rebase Into Current Queue Options (recommended)
    Resolve the current main conflicts by carrying the operation-local marker through the existing ReplyBackendQueueMessageOptions path and rerun focused delivery tests.
  2. Accept Focused Proof Without Live WhatsApp
    Maintainers may land after source and focused tests if they accept that transport-level WhatsApp media proof is not required for this narrow metadata handoff.
  3. Pause For Fresh Replacement Branch
    If the conflict resolution becomes broader than the original patch, keep the linked issue open and replace this branch with a smaller current-main PR.

Next step before merge

  • [P2] This protected MEMBER-authored PR has no narrow automated repair target right now; the next action is maintainer rebase/review of the current-main conflict, delivery risk, and proof bar.

Maintainer decision needed

  • Question: Should this maintainer-labeled PR be rebased and landed as the canonical operation-local fix for steered inbound-audio TTS?
  • Rationale: The bug remains source-reproducible, but the branch now conflicts with current main and changes final media modality, so automation should not choose merge timing or proof expectations for the assigned maintainer.
  • Likely owner: steipete — steipete is the live assignee and has adjacent merger history for inbound-audio TTS propagation.
  • Options:
    • Rebase And Land Operation-Local Fix (recommended): Adapt the marker to the current queue-message option type, rerun focused merge-head validation, and keep this as the canonical fix for the linked issue.
    • Require Live WhatsApp Proof First: Ask for a redacted active-steer WhatsApp audio inbound showing the final reply delivered as voice media before merge.
    • Pause If Conflict Rewrite Changes Shape: If the rebase requires a broader state-boundary rewrite, pause this PR and keep the linked issue open for a fresh focused branch.

Security
Cleared: The diff changes TypeScript runtime metadata and tests only; it does not touch dependencies, workflows, permissions, secrets, lockfiles, install scripts, or publishing surfaces.

Review details

Best possible solution:

Rebase and adapt the operation-local marker to current main’s queue option shape, then land it as the canonical fix if merge-head validation and any maintainer-required WhatsApp proof pass.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection gives a high-confidence reproduction path: a text-originated active run can accept an audio follow-up, but current main does not carry that audio fact into final TTS dispatch. I did not run a live WhatsApp timing repro in this read-only review.

Is this the best way to solve the issue?

Yes, the operation-local marker remains the narrowest maintainable fix shape because the audio fact belongs to the accepted reply operation, not persisted session state. The branch still needs rebase/adaptation before it can be merged on current main.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 118e5bd76223.

Label changes

Label changes:

  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.
  • remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

  • P2: This is a normal-priority WhatsApp/TTS delivery bug with real but bounded user-visible impact.
  • merge-risk: 🚨 message-delivery: The PR changes whether accepted active steer-mode replies are delivered as plain text or voice media when inbound-auto TTS is enabled.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate is not applicable to this MEMBER-authored maintainer PR; the body lists focused command proof, while live WhatsApp media proof remains optional maintainer risk reduction.
Evidence reviewed

PR surface:

Source +28, Tests +109. Total +137 across 12 files.

View PR surface stats
Area Files Added Removed Net
Source 6 34 6 +28
Tests 6 112 3 +109
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 12 146 9 +137

What I checked:

  • Repository policy read: Read the full root AGENTS.md plus scoped src/agents and embedded-runner guidance; the review applied whole-path PR review, protected maintainer handling, and message-delivery risk policy. (AGENTS.md:1, 118e5bd76223)
  • Live PR state: Live GitHub state shows the PR is open, authored by mcaxtr, assigned to steipete, labeled maintainer, and mergeStateStatus is DIRTY/CONFLICTING against main. (1a96d268f0f0)
  • Current main still drops the audio fact at active steering: Current main queues active steer-mode prompts with steering/debounce/transcript-recorder options, but does not pass followupRun.currentInboundAudio. (src/auto-reply/reply/agent-runner.ts:1267, 118e5bd76223)
  • Current main still gates final TTS from original context: Current main computes inboundAudio from the dispatch ctx and passes that value into final TTS, which remains false for a text-originated run later steered by audio. (src/auto-reply/reply/dispatch-from-config.ts:1454, 118e5bd76223)
  • TTS contract requires inboundAudio for inbound-auto synthesis: The speech runtime returns without synthesis when autoMode is inbound and params.inboundAudio is not true. (packages/speech-core/src/tts.ts:2003, 118e5bd76223)
  • PR patch carries the operation-local marker: The PR diff adds currentInboundAudio to the steer queue options, marks the reply operation after active-session steering accepts it, and resolves final inboundAudio from the operation marker. (src/auto-reply/reply/agent-runner.ts:1287, 1a96d268f0f0)

Likely related people:

  • steipete: Live PR state assigns this item to steipete, and PR 87749 shows steipete merged adjacent inbound-audio TTS propagation work. (role: assigned reviewer and adjacent merger; confidence: high; commits: eb417bc672e6; files: src/agents/tools/message-tool.ts, src/infra/outbound/message-action-runner.ts, src/auto-reply/reply/agent-runner-utils.ts)
  • mcaxtr: Beyond authoring this PR, GitHub commit search shows prior merged WhatsApp and message-delivery fixes from mcaxtr, including WhatsApp streaming and voice-message MIME work. (role: current PR author and adjacent WhatsApp/message-delivery contributor; confidence: medium; commits: 1a96d268f0f0, d2d9a928b1d2, 186dc0363fb6; files: src/auto-reply/reply/agent-runner.ts, src/auto-reply/reply/dispatch-from-config.ts, src/auto-reply/reply/reply-run-registry.ts)
  • ai-hpc: Authored merged PR 87749, which carried inbound-audio context through sibling message-tool TTS paths but did not cover this active-run final-dispatch handoff. (role: adjacent inbound-audio metadata contributor; confidence: medium; commits: eb417bc672e6; files: src/agents/tools/message-tool.ts, src/infra/outbound/message-action-runner.ts, src/auto-reply/reply/agent-runner-utils.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (1 earlier review cycle)
  • reviewed 2026-06-23T17:58:09.093Z sha 1a96d26 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 21, 2026
@mcaxtr
mcaxtr force-pushed the fix/pr-94049-steered-inbound-audio branch from aaafe08 to b0561e4 Compare June 21, 2026 19:08
@xuwei-xy

xuwei-xy commented Jun 21, 2026

Copy link
Copy Markdown

Does this change affect the behavior of config in any way?

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 21, 2026
@totobusnello totobusnello mentioned this pull request Jun 21, 2026
Merged
3 tasks
@mcaxtr
mcaxtr force-pushed the fix/pr-94049-steered-inbound-audio branch from b0561e4 to 1a96d26 Compare June 23, 2026 01:49
This was referenced Jun 24, 2026
Closed
Closed
@steipete steipete self-assigned this Jul 7, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 7, 2026
@steipete
steipete force-pushed the fix/pr-94049-steered-inbound-audio branch from 1a96d26 to 30eabab Compare July 7, 2026 04:08
@steipete

steipete commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Maintainer prep complete for 30eababa41d.

  • Reworked the fix around one operation-local fact, latched only after an active run accepts steered audio.
  • Removed the embedded queue-option plumbing and durable/session-state alternatives.
  • Covered both automatic final/block/tool TTS and message_tool_only delivery with late reads of the same operation state.
  • Blacksmith Testbox: 382 focused tests passed; exact rebased pnpm check:changed passed core/test typechecks, every core lint shard, and runtime guards.
  • Fresh autoreview: clean, 0.91 confidence.
  • Exact-head CI: https://github.com/openclaw/openclaw/actions/runs/28840803266

Known proof gap: no separate credentialed WhatsApp roundtrip was available; the contributor issue reproduction plus dispatch/message-tool regression coverage exercises the affected ownership paths.

@steipete
steipete merged commit 31432bf into main Jul 7, 2026
108 checks passed
@steipete
steipete deleted the fix/pr-94049-steered-inbound-audio branch July 7, 2026 04:18
@steipete

steipete commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

@clawsweeper clawsweeper Bot mentioned this pull request Jul 7, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 7, 2026
@mcaxtr @steipete
fix(reply): preserve steered audio for inbound TTS (openclaw#95596)
5da10d1 
Co-authored-by: Peter Steinberger <[email protected]>
sheyanmin pushed a commit to sheyanmin/openclaw that referenced this pull request Jul 8, 2026
@mcaxtr @steipete @sheyanmin
fix(reply): preserve steered audio for inbound TTS (openclaw#95596)
d3fd5b1 
Co-authored-by: Peter Steinberger <[email protected]>
giodl73-repo pushed a commit to giodl73-repo/openclaw that referenced this pull request Jul 8, 2026
@mcaxtr @steipete @giodl73-repo
fix(reply): preserve steered audio for inbound TTS (openclaw#95596)
a147c92 
Co-authored-by: Peter Steinberger <[email protected]>
@clawsweeper clawsweeper Bot mentioned this pull request Jul 9, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@steipete steipete

Labels

agents Agent runtime and tooling maintainer Maintainer-authored PR merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] WhatsApp + tts.auto: "inbound": voice reply is delivered as plain text when an audio inbound arrives during an active session run

3 participants

@mcaxtr @xuwei-xy @steipete