Skip to content

WhatsApp: add preflight audio transcription for DM voice notes#64120

Merged
mcaxtr merged 6 commits into
openclaw:mainfrom
rogerdigital:fix/whatsapp-dm-audio-preflight-transcription
Apr 25, 2026
Merged

WhatsApp: add preflight audio transcription for DM voice notes#64120
mcaxtr merged 6 commits into
openclaw:mainfrom
rogerdigital:fix/whatsapp-dm-audio-preflight-transcription

Conversation

@rogerdigital

@rogerdigital rogerdigital commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a WhatsApp regression where DM voice notes reached the agent as raw <media:audio> placeholders with no preflight STT attempt, even when audio transcription was configured.

This PR restores the missing preflight transcription step so the inbound context carries spoken text instead of the placeholder when transcription succeeds. Failures remain non-fatal and fall back to the original placeholder.

What Changed

  • added a lazy WhatsApp audio-preflight runtime boundary
  • transcribed WhatsApp voice notes before inbound context construction
  • made command detection/auth use the transcript body when preflight succeeds
  • deduplicated preflight STT across broadcast fan-out via preflightAudioTranscript
  • distinguished string | null | undefined preflight states so failed preflight does not retry per agent
  • moved ack sending ahead of slow STT work for voice notes
  • gated early DM audio preflight behind access control
  • preserved per-agent ack checks during group broadcast fan-out
  • merged the latest main changes, including the newer groupSystemPrompt context field

Root Cause

WhatsApp inbound processing built the agent-facing inbound line/context directly from msg.body, which for voice notes is always <media:audio>. Unlike the existing Telegram and Discord paths, there was no WhatsApp preflight call to transcribeFirstAudio(...) before dispatch.

Scope

Touched files are limited to the WhatsApp inbound/monitor path:

  • extensions/whatsapp/src/auto-reply/monitor/audio-preflight.runtime.ts
  • extensions/whatsapp/src/auto-reply/monitor/broadcast.ts
  • extensions/whatsapp/src/auto-reply/monitor/on-message.ts
  • extensions/whatsapp/src/auto-reply/monitor/on-message.audio-preflight.test.ts
  • extensions/whatsapp/src/auto-reply/monitor/process-message.ts
  • extensions/whatsapp/src/auto-reply/monitor/process-message.audio-preflight.test.ts
  • extensions/whatsapp/src/inbound/monitor.ts
  • extensions/whatsapp/src/inbound/types.ts
  • extensions/whatsapp/src/monitor-inbox.blocks-messages-from-unauthorized-senders-not-allowfrom.test-support.ts

No new SDK surface was added.

Verification

Focused local verification after the latest review fixes:

pnpm test extensions/whatsapp/src/auto-reply/monitor/process-message.audio-preflight.test.ts extensions/whatsapp/src/auto-reply/monitor/on-message.audio-preflight.test.ts
pnpm exec oxlint extensions/whatsapp/src/auto-reply/monitor/on-message.ts extensions/whatsapp/src/auto-reply/monitor/on-message.audio-preflight.test.ts

Latest result:

  • 2 test files passed
  • 14 tests passed
  • oxlint passed with 0 warnings / 0 errors
  • PR CI is green

Review Status

All functional Codex review items raised on this PR have been addressed and their threads resolved.

Out of Scope / Follow-up

Two broader concerns were intentionally kept out of this PR’s scope:

  • the shared cross-channel double-STT limitation after preflight
  • shared STT timeout / budgeting hardening in the media-understanding primitive

Both are better handled in shared media-understanding / SDK follow-up work rather than by expanding this WhatsApp regression fix.

Linked Issue

@openclaw-barnacle openclaw-barnacle Bot added channel: whatsapp-web Channel integration: whatsapp-web size: M labels Apr 10, 2026
@greptile-apps

greptile-apps Bot commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a preflight audio transcription step to WhatsApp DM voice note processing, mirroring the existing fix applied to Telegram in #61008. The implementation correctly follows the lazy-loading boundary pattern (audio-preflight.runtime.ts), is non-fatal on transcription failure, and is covered by 7 targeted unit tests spanning success, error, and guard-condition scenarios. No new SDK surface is added and all other channel behaviors are untouched.

Confidence Score: 5/5

This PR is safe to merge; the change is narrowly scoped, non-fatal on failure, and all remaining observations are P2 style notes.

Implementation mirrors a proven Telegram pattern exactly, follows the lazy-loading boundary convention, is guarded by three independent conditions, and is covered by 7 targeted unit tests. No P0 or P1 findings. The only minor observation (redundant ternary inside the mediaType guard) is cosmetic and correct.

No files require special attention.

Reviews (1): Last reviewed commit: "WhatsApp: add preflight audio transcript..." | Re-trigger Greptile

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2cfa68e8b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/whatsapp/src/auto-reply/monitor/process-message.ts Outdated
@rogerdigital
rogerdigital force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch from 2cfa68e to c316ae0 Compare April 10, 2026 06:05
@rogerdigital

Copy link
Copy Markdown
Contributor Author

Re: Aisle Security Analysis — Unauthorized audio transcription preflight executed before access-control gating

The preflight transcription does not execute before access-control gating. By the time processMessage is called, the message has already cleared all channel-level access controls:

  • Group messages: on-message.ts calls applyGroupGating() (line 129) before processMessage is ever reached. Messages that fail gating return early at line 145 (if (!gating.shouldProcess) return).
  • DM messages: routing and pairing-policy checks are applied at the resolveAgentRoute / channel dispatch layer before processMessage is invoked.

processMessage is the post-gating processing stage — it only handles messages that have already been admitted by the channel access model.

Additionally, the audio transcription is strictly subordinate to the existing LLM call: every message that reaches processMessage already triggers dispatchWhatsAppBufferedReply → model inference, which is a significantly larger API surface than a single STT call. If unauthorized senders were able to reach this stage, the DoS and privacy exposure from the model call would far exceed that of the preflight transcription.

No trust boundary is being crossed here. The preflight mirrors the existing behavior for Telegram (in production since #61008) and Discord, which apply the same pattern in the same post-gating position.

@rogerdigital

rogerdigital commented Apr 10, 2026

Copy link
Copy Markdown
Contributor Author

Re: Aisle Security Analysis — Round 2

Issue 2 (Medium — loose MIME check): Fixed.

Changed mediaType?.includes("audio") to mediaType?.startsWith("audio/") in the amended commit (d801b6514b). This eliminates the false-positive surface for crafted MIME strings like application/notaudio-but-contains-audio. All 8 regression tests still pass.


Issue 1 (High — no timeout/size limits): Out of scope for this PR.

The concern is valid in the abstract, but this PR is a parity fix that mirrors the existing Telegram preflight (shipped in #61008) and Discord, both of which use the same inline await transcribeFirstAudio(...) pattern without per-call timeouts or size guards. Introducing those controls here (and not there) would create an inconsistent, partially-hardened surface — the real fix would be adding resource controls to the shared transcribeFirstAudio SDK primitive itself so all channels benefit equally.

Additionally, resource limits at the media download layer (maxMediaBytes) already constrain what gets saved to mediaPath before this code is reached.

If maintainers agree that timeout/concurrency controls belong at the SDK primitive level, I'm happy to file a follow-up issue. Adding them only to the WhatsApp path in this PR would be scope creep relative to the fix goal.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d801b6514b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/whatsapp/src/auto-reply/monitor/process-message.ts
@rogerdigital
rogerdigital force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch from d801b65 to 1a57372 Compare April 10, 2026 06:49

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a57372932

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/whatsapp/src/auto-reply/monitor/process-message.ts Outdated
@rogerdigital
rogerdigital force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch from 1a57372 to c8ebbe5 Compare April 10, 2026 07:15

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8ebbe5e3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/whatsapp/src/auto-reply/monitor/process-message.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8ebbe5e3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/whatsapp/src/auto-reply/monitor/process-message.ts
@rogerdigital
rogerdigital force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch from c8ebbe5 to 6daf3d2 Compare April 10, 2026 08:50

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6daf3d2cc0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/whatsapp/src/auto-reply/monitor/process-message.ts Outdated
@rogerdigital
rogerdigital force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch 4 times, most recently from 5252071 to c2359d5 Compare April 13, 2026 05:41
@rogerdigital

Copy link
Copy Markdown
Contributor Author

Re: CI failures — all pre-existing on `main`, unrelated to this PR

The three failing checks (check, check-additional, checks-node-test) were already failing on main at the time this PR was submitted. None of the failures touch any file modified by this PR.

Check Failure cause Status on `main`
`check` TS errors in `extensions/nostr`, `extensions/tlon`, `src/gateway/server`, `ui/src/ui` — all unrelated to WhatsApp ✅ Green on `main` run #24325050280
`checks-node-test` `register.subclis.test.ts` assertion failures in CLI registration — unrelated to WhatsApp ✅ Green on `main` run #24325050280
`check-additional` `lint:tmp:no-raw-channel-fetch` + `test:extensions:package-boundary` — upstream backlog ✅ Green on `main` run #24325050280

This PR only touches five files under extensions/whatsapp/src/auto-reply/monitor/, all of which pass their own dedicated check (extension-fast-whatsapp ✅).

I've also rebased onto the latest main to keep the branch current.

@rogerdigital

Copy link
Copy Markdown
Contributor Author

Re: Aisle Security Analysis — Issue 1 (transcript in logs)

The two log lines flagged by Aisle (body: elide(combinedBody, 240) and mediaPath) are not new — they exist verbatim in the pre-PR baseline on main (process-message.ts L242–L244). This PR does not add, move, or change either log call.

What this PR does change is the value of combinedBody for audio messages: previously it was always <media:audio> (the literal placeholder), now it may contain the transcript. The log call itself is untouched.

A few additional mitigations already present:

  • elide(combinedBody, 240) caps the logged string at 240 characters regardless of transcript length.
  • mediaPath is logged the same way for every inbound message type — text, image, video, audio — before and after this PR.
  • Every message that reaches processMessage has already triggered a full LLM inference call via dispatchWhatsAppBufferedReply, which is a significantly larger surface than a single STT call. If transcript content in logs were the concern, the model call payload would be the primary vector.

If the project wants to treat transcripts as sensitive and avoid logging them by default, that is a valid improvement — but the correct fix is to audit elide/replyLogger.info across all channels (WhatsApp, Telegram, Discord) and possibly add a redaction helper to the shared logging layer. Doing it only on the WhatsApp audio path here would create an inconsistent, partially-hardened surface. I'm happy to file a follow-up issue for that work if maintainers agree it belongs at the SDK/logger level.

@rogerdigital

Copy link
Copy Markdown
Contributor Author

Re: checks-windows-node-test failure — CI infrastructure flake, unrelated to this PR

The failure occurred at the Download dist artifact step before any test code ran. The Windows runner made 5 attempts to fetch the dist-build artifact and received an HTML error page each time instead of a valid JSON response — a known symptom of a transient GitHub Actions artifact service outage.

Attempt 1 of 5 failed: Unexpected token '<', "<html>..." is not valid JSON
Attempt 2 of 5 failed: ...
(all 5 attempts failed)

The build-artifacts job completed successfully in the same run, so the artifact was correctly produced. The Windows runner simply could not reach it due to a backend blip. No test code was executed.

Requesting a re-run of the failed job.

@rogerdigital
rogerdigital force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch 3 times, most recently from a28b4ad to c2359d5 Compare April 13, 2026 13:52
@rogerdigital

Copy link
Copy Markdown
Contributor Author

Thanks — I addressed the Codex review items in follow-up amendments.

Current state:

  • preflight transcription is now computed before broadcast fan-out, so one inbound voice note does not trigger duplicate STT per target agent
  • the downstream path carries explicit preflight state via preflightAudioTranscript
  • string | null is used to distinguish “preflight attempted but no transcript” from “preflight not attempted”, avoiding retry-based duplicate transcription on failure paths

The PR remains scoped to WhatsApp DM voice-note preflight only, with no routing or group-behavior changes. Happy to revise further if you'd like this shaped differently.

@rogerdigital
rogerdigital force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch from c2359d5 to 5376cd8 Compare April 18, 2026 10:36

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5376cd87ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/whatsapp/src/auto-reply/monitor/on-message.ts
@mcaxtr mcaxtr self-assigned this Apr 25, 2026
@mcaxtr
mcaxtr force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch 2 times, most recently from 1471203 to 08dac74 Compare April 25, 2026 01:10
@mcaxtr
mcaxtr force-pushed the fix/whatsapp-dm-audio-preflight-transcription branch from 08dac74 to 7480b33 Compare April 25, 2026 01:12
@mcaxtr
mcaxtr merged commit ea168c2 into openclaw:main Apr 25, 2026
55 of 56 checks passed
@mcaxtr

mcaxtr commented Apr 25, 2026

Copy link
Copy Markdown
Member

Merged via squash.

Thanks @rogerdigital!

@vincentkoc vincentkoc added the dedupe:parent Primary canonical item in dedupe cluster label Apr 25, 2026
Angfr95 pushed a commit to Angfr95/openclaw that referenced this pull request Apr 25, 2026
…law#64120)

Merged via squash.

Prepared head SHA: 7480b33
Co-authored-by: rogerdigital <[email protected]>
Co-authored-by: mcaxtr <[email protected]>
Reviewed-by: @mcaxtr
@rogerdigital
rogerdigital deleted the fix/whatsapp-dm-audio-preflight-transcription branch April 26, 2026 09:16
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
…law#64120)

Merged via squash.

Prepared head SHA: 7480b33
Co-authored-by: rogerdigital <[email protected]>
Co-authored-by: mcaxtr <[email protected]>
Reviewed-by: @mcaxtr
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
…law#64120)

Merged via squash.

Prepared head SHA: 7480b33
Co-authored-by: rogerdigital <[email protected]>
Co-authored-by: mcaxtr <[email protected]>
Reviewed-by: @mcaxtr
globalcaos pushed a commit to globalcaos/tinkerclaw that referenced this pull request May 13, 2026
…law#64120)

Merged via squash.

Prepared head SHA: 7480b33
Co-authored-by: rogerdigital <[email protected]>
Co-authored-by: mcaxtr <[email protected]>
Reviewed-by: @mcaxtr
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
…law#64120)

Merged via squash.

Prepared head SHA: 7480b33
Co-authored-by: rogerdigital <[email protected]>
Co-authored-by: mcaxtr <[email protected]>
Reviewed-by: @mcaxtr
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
…law#64120)

Merged via squash.

Prepared head SHA: 7480b33
Co-authored-by: rogerdigital <[email protected]>
Co-authored-by: mcaxtr <[email protected]>
Reviewed-by: @mcaxtr
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
…law#64120)

Merged via squash.

Prepared head SHA: 7480b33
Co-authored-by: rogerdigital <[email protected]>
Co-authored-by: mcaxtr <[email protected]>
Reviewed-by: @mcaxtr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: whatsapp-web Channel integration: whatsapp-web dedupe:parent Primary canonical item in dedupe cluster size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: WhatsApp DM voice notes bypass STT pipeline entirely — audio arrives as raw <media:audio> with no transcription attempt

3 participants