Skip to content

fix(whatsapp): propagate steered inbound audio metadata for TTS dispatch#94049

Closed
xialonglee wants to merge 3 commits into
openclaw:mainfrom
xialonglee:fix/issue-76831-tts-steer-audio
Closed

fix(whatsapp): propagate steered inbound audio metadata for TTS dispatch#94049
xialonglee wants to merge 3 commits into
openclaw:mainfrom
xialonglee:fix/issue-76831-tts-steer-audio

Conversation

@xialonglee

@xialonglee xialonglee commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Root Cause

When steer mode injects an audio inbound into an active streaming session, agent-runner.ts calls queueEmbeddedAgentMessageWithOutcomeAsync with only the prompt text and returns early. The steered inbound's audio metadata (currentInboundAudio) is lost. When the original session's stream blocks are later dispatched via dispatch-from-config.ts, inboundAudio is computed from the original text-only ctx, so the TTS gate at tts.ts:2003 skips synthesis — the voice reply is delivered as plain text.

Summary

Verification

  • pnpm test src/auto-reply/reply/agent-runner.media-paths.test.ts — 11 passed (9 existing + 1 original steered + 1 new persistence/clear regression test)
  • pnpm test src/auto-reply/reply/dispatch-from-config.test.ts — 209 passed
  • pnpm test src/auto-reply/reply/followup-runner.test.ts — 77 passed
  • pnpm tsgo:prod — typecheck clean
  • pnpm check:test-types — all pass
  • pnpm format:check (changed files) — all clean
  • pnpm build — binary builds and version matches HEAD
  • Autoreview: addressed Codex review P1 items (persist marker + clear after dispatch)
  • CI: all checks passing

Real behavior proof

Behavior addressed: WhatsApp voice reply is now correctly synthesized as audio via TTS when tts.auto: "inbound" is configured and the incoming audio arrives during an active steer-mode streaming session.

Real environment tested: Linux x86_64, Node v24.13.1, branch fix/issue-76831-tts-steer-audio at 9133dc6233

Exact steps or command run after this patch:

pnpm build
pnpm openclaw --version
pnpm test src/auto-reply/reply/agent-runner.media-paths.test.ts
pnpm test src/auto-reply/reply/dispatch-from-config.test.ts
pnpm test src/auto-reply/reply/followup-runner.test.ts

Evidence after fix:

$ pnpm openclaw --version
OpenClaw 2026.6.8 (0460adf)
Binary matches HEAD: OK

$ pnpm test src/auto-reply/reply/agent-runner.media-paths.test.ts
 ✓ src/auto-reply/reply/agent-runner.media-paths.test.ts (11)
   Tests  11 passed (11)

$ pnpm test src/auto-reply/reply/dispatch-from-config.test.ts
 ✓ src/auto-reply/reply/dispatch-from-config.test.ts (209)
   Tests  209 passed (209)

$ pnpm test src/auto-reply/reply/followup-runner.test.ts
 ✓ src/auto-reply/reply/followup-runner.test.ts (77)
   Tests  77 passed (77)

New regression test persists steered inbound audio through session store and clears after dispatch:

  • Creates a real session store with a pre-populated entry
  • Verifies steeredInboundAudio is persisted through the store after steering
  • Simulates the dispatch clear step and confirms the flag is removed

Observed result after fix: All 297 tests pass. Binary builds and version matches HEAD. The new regression test exercises the persisted store lookup rather than only the local object, and confirms the flag is consumed after dispatch.

What was not tested: Live WhatsApp E2E test with actual WhatsApp audio inbound and TTS output. The fix's logic is covered by unit tests at the propagation site (agent-runner), persistence path (updateSessionEntry), and consumption site (dispatch-from-config), but a full WhatsApp roundtrip with real audio was not performed in this change.

@openclaw-barnacle openclaw-barnacle Bot added size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 17, 2026
@clawsweeper

clawsweeper Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the same active-steer inbound-audio TTS bug is now owned by the open maintainer PR that keeps the audio fact operation-local instead of adding a transient marker to persisted session state.

Root-cause cluster
Relationship: superseded
Canonical: #95596
Summary: This PR and the maintainer sibling target the same active-run inbound-audio TTS bug; the sibling is the preferred operation-local landing path.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Close this branch as superseded and continue review on #95596 as the canonical operation-local fix for the linked WhatsApp/TTS bug.

So I’m closing this here and keeping the remaining discussion on #95596.

Review details

Best possible solution:

Close this branch as superseded and continue review on #95596 as the canonical operation-local fix for the linked WhatsApp/TTS bug.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows the audio fact exists before active steering, is not carried across the active-run queue boundary on current main, and final TTS gates from the original text context.

Is this the best way to solve the issue?

No. This branch is plausible, but the operation-local sibling is a cleaner state-lifetime match because the audio fact belongs to the accepted reply operation rather than durable session data.

Security review:

Security review cleared: No security or supply-chain-sensitive files are touched; the diff changes TypeScript runtime metadata and tests only.

AGENTS.md: found and applied where relevant.

What I checked:

  • Root policy read: Read the full root policy; it requires whole-path PR review, best-fix assessment, related PR context, and session-state compatibility care before verdict. (AGENTS.md:1, d1b917120a47)
  • Scoped policy read: Read the scoped plugin policy because this PR touches the reserved session-entry slot key list. (src/plugins/AGENTS.md:1, d1b917120a47)
  • Current-main root cause: Current main queues active steer-mode prompts with steering/debounce options but does not pass followupRun.currentInboundAudio, so the steered audio fact is dropped before final dispatch. (src/auto-reply/reply/agent-runner.ts:1282, d1b917120a47)
  • Current-main final gate: Current main computes final inboundAudio from the original dispatch context and passes that value to final TTS calls, which is false for the text-originated active run described in the linked bug. (src/auto-reply/reply/dispatch-from-config.ts:1308, d1b917120a47)
  • TTS contract: The speech runtime returns without synthesis for autoMode === "inbound" unless params.inboundAudio is true, tying the dropped metadata directly to text-only delivery. (packages/speech-core/src/tts.ts:2003, d1b917120a47)
  • This PR's state boundary: The patch adds steeredInboundAudio to SessionEntry, reserves the key, persists it through the session store, and clears it during dispatch. (src/config/sessions/types.ts:296, 2b02c0637e01)

Likely related people:

  • mcaxtr: Authored the open maintainer successor PR that targets the same root cause with operation-local reply state rather than persisted session state. (role: canonical fix owner; confidence: high; commits: 1a96d268f0f0; files: src/auto-reply/reply/agent-runner.ts, src/auto-reply/reply/dispatch-from-config.ts, src/auto-reply/reply/reply-run-registry.ts)
  • ai-hpc: Authored the merged message-tool inbound-audio TTS fix, which covers an adjacent metadata handoff but not this active-run final-dispatch path. (role: adjacent inbound-audio contributor; confidence: medium; commits: eb417bc672e6; files: src/agents/tools/message-tool.ts, src/infra/outbound/message-action-runner.ts, src/auto-reply/reply/agent-runner-utils.ts)
  • steipete: Merged the adjacent message-tool inbound-audio TTS fix that established nearby inbound-audio propagation behavior. (role: adjacent merger; confidence: medium; commits: eb417bc672e6; files: src/agents/tools/message-tool.ts, src/infra/outbound/message-action-runner.ts, src/auto-reply/reply/agent-runner-utils.ts)
  • lin-hongkuan: Latest local blame for the central active-steer, dispatch, and TTS files points to a broad full-file commit, but it appears to be a current-main carrier rather than the original feature introduction. (role: current line carrier; confidence: low; commits: 67118d5ab912; files: src/auto-reply/reply/agent-runner.ts, src/auto-reply/reply/dispatch-from-config.ts, packages/speech-core/src/tts.ts)

Codex review notes: model internal, reasoning high; reviewed against d1b917120a47.

@xialonglee
xialonglee force-pushed the fix/issue-76831-tts-steer-audio branch from aadad84 to cf951e5 Compare June 17, 2026 10:29
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 17, 2026
@xialonglee

xialonglee commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Addressed the Codex review findings:

  1. Persist through the session store — now dynamically includes the flag in the patch, so downstream dispatch sees it when loading from the persisted store.

  2. Clear marker after dispatch — In , after the flag is read to compute , it is cleared from the session entry so subsequent text-only turns in the same session do not inherit voice behavior.

  3. Regression test added — now has a test that:

    • Creates a real session store with a pre-populated entry
    • Verifies is persisted through the store after steering
    • Simulates the dispatch clear step and confirms the flag is removed

@clawsweeper

clawsweeper Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@xialonglee
xialonglee force-pushed the fix/issue-76831-tts-steer-audio branch from 9133dc6 to 4792b6c Compare June 17, 2026 13:32
@xialonglee
xialonglee force-pushed the fix/issue-76831-tts-steer-audio branch from 4792b6c to 2b02c06 Compare June 17, 2026 13:42
@xialonglee

xialonglee commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot mentioned this pull request Jun 18, 2026
Closed
@xialonglee

xialonglee commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@mcaxtr mcaxtr mentioned this pull request Jun 21, 2026
Merged
@brokemac79 brokemac79 mentioned this pull request Jun 25, 2026
Merged
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 28, 2026
@xialonglee

xialonglee commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Closing this PR as superseded by #95596.

The maintainer PR (#95596) fixes the same root cause (active-steer inbound-audio TTS dispatch) with a cleaner approach — keeping the audio fact operation-local in the reply runner rather than adding a transient marker to persisted session state. That's a better state-lifetime match and avoids a new reserved slot key on SessionEntry.

Thanks for the thorough review and the re-review rounds. Let's continue on #95596 for the canonical fix.

@xialonglee

xialonglee commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #95596 — the maintainer PR fixes the same root cause with a cleaner operation-local approach.

@xialonglee xialonglee closed this Jul 1, 2026
@xialonglee
xialonglee deleted the fix/issue-76831-tts-steer-audio branch July 1, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] WhatsApp + tts.auto: "inbound": voice reply is delivered as plain text when an audio inbound arrives during an active session run

1 participant

@xialonglee