Skip to content

[Bug] WhatsApp + tts.auto: "inbound": voice reply is delivered as plain text when an audio inbound arrives during an active session run #76831

Description

@aleps001

Summary

On WhatsApp, messages.tts.auto: "inbound" is expected to deliver replies as voice when the inbound message is audio. In specific timing conditions (a second inbound arriving while a prior run is still active), the reply content is correctly generated for the audio question, but the reply is delivered as plain text with no TTS synthesis attempt. The same agent on the same session correctly emits voice for other audio messages a few minutes later — so the channel/provider side is healthy.

Likely the same class of dispatch-layer gate gap that #72034 (Feishu) fixed for ACP dispatch but not for the default dispatch path used by WhatsApp.

Environment

  • OpenClaw 2026.5.2 (commit 8b2a6e5), Linux x64 (WSL2 Ubuntu 24.04)
  • Channel: WhatsApp (Baileys/whatsapp provider)
  • Agent runtime: agentRuntime.id=codex (native Codex harness, fallback=none), model openai/gpt-5.5
  • TTS config: messages.tts.provider=elevenlabs, tts.auto set to "inbound" mode (default)
  • messages.queue not configured → defaults apply (mode=steer, debounceMs=500)
  • channels.whatsapp.debounceMs = 0

Reproduction (observed once, real session)

Single user, single direct WhatsApp DM, two inbound messages 8 seconds apart while a long turn is in flight:

t (BRT) inbound media correlationId
10:45:32 text "L" none 3A6235D61D030F1992CC
10:45:40 audio "Como foi meu sono essa noite?" (transcribed) audio/ogg; codecs=opus 3ADCFF7B13FB361B2A20
10:47:35 outbound reply answering the sleep question text only (no mediaUrl, mediaKind=null) 3A6235D61D030F1992CC (the text message's ID)

Subsequent control: 11:12:15, another audio inbound from the same user → reply correctly delivered as mediaUrl=/home/aleps/.openclaw/media/outbound/voice-…ogg, mediaKind=audio. So the TTS plumbing works in steady state.

What I expected

With tts.auto="inbound" and one of the two inbound messages being audio, the reply (which clearly answered the audio's question — sleep summary, not a response to "L") should have been synthesized as voice and delivered with audio media.

What seems to happen

Two candidate mechanisms, both compatible with the evidence:

(a) Dispatch path missing the inboundAudio gate

In dist/dispatch-qvEfkoBF.js (default dispatch — used by WhatsApp via monitor-*.js), maybeApplyTtsToReplyPayload does not check params.inboundAudio even though it receives the value:

async function maybeApplyTtsToReplyPayload(params) {
    if (!shouldAttemptTtsPayload({
        cfg: params.cfg,
        ttsAuto: params.ttsAuto,
        agentId: params.agentId,
        channelId: params.channel,
        accountId: params.accountId
    })) return params.payload;
    const { maybeApplyTtsToPayload } = await loadTtsRuntime();
    return maybeApplyTtsToPayload(params);
}

Compare dist/dispatch-acp-CPEbTBot.js:500 (ACP path) where the gate is explicit:

if (ttsStatus.autoMode === "inbound" && !params.inboundAudio) return params.payload;

The closing comment on #72034 stated the runtime performs the gate after the dispatch passes inboundAudio through. If the runtime indeed gates correctly, this is not the root cause — but the asymmetry in dispatch wrappers between ACP and default is at least a code smell worth verifying.

(b) inboundAudio / entry.ttsAuto resolved against the wrong inbound under messages.queue.mode=steer (default)

At dispatch time, dispatch-qvEfkoBF.js:315-316:

const inboundAudio = isInboundAudioContext(ctx);
const sessionTtsAuto = normalizeTtsAutoMode(sessionStoreEntry.entry?.ttsAuto);

When the audio inbound is steered into an already-running turn that originated from the text message "L", the ctx and sessionStoreEntry.entry consumed at the moment of dispatch may correspond to the leading inbound (text), not the steer-injected one (audio). Result:

  • inboundAudio = false (because ctx.MediaType is text)
  • entry.ttsAuto = false (computed from L)
  • The TTS gate denies, even though the assistant content was generated against the audio question.

This would also explain why the outbound correlationId matches the text message — the dispatch envelope reflects the run originator, not the steered message.

Why I think (b) is the dominant cause

The reply content is unambiguously about sleep ("Como foi meu sono essa noite?"), so the audio's transcription was ingested into the model context — i.e. steer mode worked at the prompt level. But the dispatch metadata (correlationId, mediaKind, ttsAuto) is locked to the first entry of the run. So the problem is the replyOrigin / ttsAuto carried forward to dispatch is the run's first inbound, not the most recent or most-relevant.

If true, the right fix is for messages.queue.mode=steer to update the run's "active replyOrigin" (the source of correlationId + MediaType + ttsAuto) to the most recent steered inbound, or to track inboundAudio as a sticky-OR over all inbounds consumed during the run.

What works (rules out other causes)

  • Audio download + transcription works (response content proves audio was understood).
  • TTS provider/account works (next audio inbound 25 minutes later was synthesized correctly).
  • No errors in dispatch logs (silent miss, not a TTS synthesis failure).
  • Channel is in pairing / dmPolicy=pairing so DM routing is fine.

Suggested fix paths (non-prescriptive)

  1. Mirror the ACP-side inboundAudio gate inside maybeApplyTtsToReplyPayload in default dispatch, or assert via test that the runtime gate behaves identically on both paths.
  2. When messages.queue.mode=steer consumes a new inbound during an active run, propagate the new inbound's media metadata (MediaType, MediaPaths, computed inboundAudio, correlationId) into the run's reply-origin record before the next dispatch boundary.
  3. Optionally make entry.ttsAuto a sticky-OR within a steered run window (any audio in the consumed inbounds → ttsAuto=true for the next assistant boundary).

Workarounds tested / available

None (issue not mitigated). Increasing channels.whatsapp.debounceMs would reduce the race window but does not fix the steer-mode envelope-mismatch root cause.

Reproducer hint

Two DMs to the same WhatsApp DM, 5–10 seconds apart, with the agent already running a non-trivial turn (>30 s) when the second arrives, where the second is audio and the first is text. Repeat ~5 times to hit the race window reliably.


Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions