fix: add placeholder transcript for silent voice notes by eulicesl · Pull Request #49131 · openclaw/openclaw

eulicesl · 2026-03-17T16:26:28Z

Summary

Problem: tiny or silent voice notes skipped as tooSmall left no transcript signal in downstream context.
Why it matters: the agent could respond as if context were missing or ambiguous instead of correctly stating that the voice note was empty/silent.
What changed: when audio is skipped with the tooSmall reason, applyMediaUnderstanding() now emits a synthetic placeholder transcription and preserves the existing formatting/body flow.
What did NOT change (scope boundary): no provider call behavior changed for normal-sized audio, and no speech model/network behavior was expanded.
AI-assisted: yes (AI-assisted implementation, contributor-reviewed).
Testing level: targeted local validation; full pnpm test was not fully green on upstream main in that environment.

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

Closes Silent/empty voice notes should inject a clear message instead of appearing as an API error #48944
Related #
This PR fixes a bug or regression

Root Cause (if applicable)

Root cause: the tooSmall skip path intentionally bypassed provider transcription but did not emit any synthetic transcript/body signal, leaving downstream context unaware that the attachment was intentionally skipped because it was empty/silent.
Missing detection / guardrail: tests covered skip behavior but did not fully lock in downstream transcript/body behavior for the tiny-audio path.
Prior context (git blame, prior PR, issue, or refactor if known): reported in Silent/empty voice notes should inject a clear message instead of appearing as an API error #48944.
Why this regressed now: the optimization to skip tiny audio avoided wasted provider calls, but the user-facing/context-facing fallback message was never added.
If unknown, what was ruled out: the issue was not a provider transcription quality problem; it occurred before provider transcription was ever attempted.

Regression Test Plan (if applicable)

Coverage level that should have caught this:
- Unit test
- Seam / integration test
- End-to-end test
- Existing coverage already sufficient
Target test or file:
- src/media-understanding/apply.test.ts
- src/media-understanding/format.test.ts
- src/media-understanding/runner.skip-tiny-audio.test.ts
Scenario the test should lock in: a tiny/silent audio attachment skipped as tooSmall should still yield deterministic transcript/body context explaining silence/emptiness.
Why this is the smallest reliable guardrail: the bug is in the local media-understanding fallback path, so unit tests can cover it directly without a live transcription provider.
Existing test that already covers this (if any): skip-tiny-audio coverage existed, but it was not sufficient to assert the final transcript/body behavior.
If no new test is added, why not: N/A.

User-visible / Behavior Changes

Silent or empty voice notes now produce a placeholder transcript instead of a confusing blank context gap.
Downstream responses can accurately say the note was empty/silent.

Diagram (if applicable)

N/A

Security Impact (required)

New permissions/capabilities? (Yes/No): No
Secrets/tokens handling changed? (Yes/No): No
New/changed network calls? (Yes/No): No
Command/tool execution surface changed? (Yes/No): No
Data access scope changed? (Yes/No): No
If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

OS: local macOS dev environment
Runtime/container: repo-local Node/pnpm test environment
Model/provider: N/A (skip path does not call provider)
Integration/channel (if any): media-understanding pipeline
Relevant config (redacted): default local test env

Steps

Feed a tiny or silent voice note that falls below MIN_AUDIO_FILE_BYTES.
Let media-understanding process the attachment.
Inspect the resulting transcript/body context.

Expected

The attachment is still skipped for provider transcription.
A deterministic placeholder transcript is emitted explaining that the note was empty/silent.

Actual

Matched expected in targeted tests after this patch.

Evidence

Attach at least one:

Failing test/log before + passing after
Trace/log snippets
Screenshot/recording
Perf numbers (if relevant)
Targeted validation run:
- pnpm build ✅
- pnpm check ✅
- src/media-understanding/runner.skip-tiny-audio.test.ts ✅
- src/media-understanding/apply.test.ts ✅
- src/media-understanding/format.test.ts ✅

Human Verification (required)

Verified scenarios:
- placeholder transcript emitted for tooSmall path
- formatting/body flow still works with synthetic transcript output
- URL-only tiny-audio path gets deterministic context
Edge cases checked:
- normal-sized audio behavior left alone
- skip path stays provider-free
What you did not verify:
- codex review --base origin/main in that environment (CLI unavailable there)
- a fully green pnpm test run in that prior environment because unrelated upstream failures reproduced independently on clean main

Review Conversations

I replied to or resolved every bot review conversation I addressed in this PR.
I left unresolved only the conversations that still need reviewer or maintainer judgment.

If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

Backward compatible? (Yes/No): Yes
Config/env changes? (Yes/No): No
Migration needed? (Yes/No): No
If yes, exact upgrade steps: N/A

Risks and Mitigations

Risk: synthetic placeholder text could surface in the wrong audio path.
- Mitigation: the fallback is explicitly limited to the tooSmall skip reason and covered by targeted tests.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3113cfeba6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

greptile-apps · 2026-03-17T16:33:16Z

Greptile Summary

This PR fixes a downstream agent context gap by injecting a synthetic audio.transcription output ([Voice note was empty or contained only silence — no speech detected]) whenever an audio attachment is skipped with a tooSmall reason, preserving the existing transcript-formatting and body-update flow without calling any provider.

Key changes

apply.ts: adds buildSyntheticSkippedAudioOutputs, which walks audio-capability attachment decisions and emits a placeholder output for any attempt whose reason starts with "tooSmall" (consistent with the "tooSmall: <message>" format written by the runner).
apply.test.ts: updates the existing URL-only tiny-audio test to assert the new placeholder behavior instead of a bare skip.

Issues found

The guard condition !outputs.some((output) => output.kind === "audio.transcription") in apply.ts is all-or-nothing: in multi-attachment scenarios where at least one audio attachment is transcribed successfully and a second is too small, buildSyntheticSkippedAudioOutputs is never called and the silent note receives no placeholder. The buildSyntheticSkippedAudioOutputs helper already iterates over individual attachments with flatMap, so the intent was clearly per-attachment; the outer guard undermines that.
Test coverage only addresses URL-only tiny audio; there is no apply-level test for path-based tiny audio files below MIN_AUDIO_FILE_BYTES.

Confidence Score: 3/5

Safe to merge for the common single-attachment voice-note case, but contains a logic gap that silently drops synthetic placeholders in multi-attachment mixed scenarios.
The core fix is correct and well-tested for the primary use case (single tiny audio attachment). The guard condition bug is a real logic issue that would cause missing agent context when a message contains multiple audio attachments of mixed sizes, which undermines the stated goal of providing deterministic context. The test coverage gap for path-based tiny audio is a secondary concern.
src/media-understanding/apply.ts — the all-or-nothing guard on line 543 needs to be made per-attachment-index aware.

Comments Outside Diff (1)

src/media-understanding/apply.test.ts, line 421-426 (link)

Test only covers URL-only tiny audio

The new test exercises the tooSmall synthetic-placeholder path via a remote URL fetch (MediaUrl). Local-path tiny audio (where MediaPath points to a small file) goes through a different code branch in the attachment cache but ultimately hits the same assertMinAudioSize guard. There is no apply-level test confirming the placeholder is also injected for path-based tiny audio, leaving a coverage gap for the most common case on self-hosted deployments where audio is delivered as a local file.

Consider adding a companion test case that sets MediaPath to a Buffer.alloc(100) fixture file instead of MediaUrl.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/media-understanding/apply.test.ts
Line: 421-426

Comment:
**Test only covers URL-only tiny audio**

The new test exercises the `tooSmall` synthetic-placeholder path via a remote URL fetch (`MediaUrl`). Local-path tiny audio (where `MediaPath` points to a small file) goes through a different code branch in the attachment cache but ultimately hits the same `assertMinAudioSize` guard. There is no `apply`-level test confirming the placeholder is also injected for path-based tiny audio, leaving a coverage gap for the most common case on self-hosted deployments where audio is delivered as a local file.

Consider adding a companion test case that sets `MediaPath` to a `Buffer.alloc(100)` fixture file instead of `MediaUrl`.

How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: src/media-understanding/apply.ts
Line: 543-545

Comment:
**Guard condition silences synthetic outputs when any transcription succeeds**

The all-or-nothing guard `!outputs.some((output) => output.kind === "audio.transcription")` means that in a multi-attachment scenario where at least one audio file is transcribed successfully and another is too small, `buildSyntheticSkippedAudioOutputs` is never called and the tiny attachment silently receives no placeholder. The function already uses `flatMap` over attachments to support multiple tiny files, but the outer guard negates that intent.

For example, if a message contains two voice notes — one real and one silent — the agent's context would include only the real transcript; the silent note's `[Voice note was empty or contained only silence…]` placeholder would be dropped entirely.

A safer approach is to filter out already-covered attachment indexes before injecting synthetics:

```ts
// Only synthesize placeholders for attachment indexes that have no real output yet
const coveredIndexes = new Set(
  outputs
    .filter((o) => o.kind === "audio.transcription")
    .map((o) => o.attachmentIndex),
);
const synthetic = buildSyntheticSkippedAudioOutputs(decisions).filter(
  (o) => !coveredIndexes.has(o.attachmentIndex),
);
outputs.push(...synthetic);
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/media-understanding/apply.test.ts
Line: 421-426

Comment:
**Test only covers URL-only tiny audio**

The new test exercises the `tooSmall` synthetic-placeholder path via a remote URL fetch (`MediaUrl`). Local-path tiny audio (where `MediaPath` points to a small file) goes through a different code branch in the attachment cache but ultimately hits the same `assertMinAudioSize` guard. There is no `apply`-level test confirming the placeholder is also injected for path-based tiny audio, leaving a coverage gap for the most common case on self-hosted deployments where audio is delivered as a local file.

Consider adding a companion test case that sets `MediaPath` to a `Buffer.alloc(100)` fixture file instead of `MediaUrl`.

How can I resolve this? If you propose a fix, please make it concise.

_{Last reviewed commit: 3113cfe}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a737a2c21b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b6d9b176d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 070412be11

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e66bbe2b49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

eulicesl · 2026-03-18T06:04:18Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e66bbe2b49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 14f2c918b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a538e72b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6648e77bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4a19af2c9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 26fed09401

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac33bd75b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…capability and prefer ordering Replace the global outputs.sort() with a targeted merge that: 1. Only sorts within the audio output slice (real + synthetic), preserving CAPABILITY_ORDER and per-capability attachments.prefer ordering for non-audio outputs. 2. Excludes synthetic placeholder indexes from audioAttachmentIndexes used by extractFileBlocks, so tiny audio-MIME files with text extensions can still be recovered via forcedTextMime. Adds mergeAudioOutputsPreservingAttachmentOrder helper.

… placeholders - merge synthetic skipped-audio placeholders using audio decision order instead of raw attachmentIndex sorting, preserving attachments.prefer - insert synthetic-only audio outputs at the audio capability slot (before video) when no real audio outputs were produced

Clarify that this synthetic transcript path is triggered by attachment size, not by a silence/no-speech detection result.

…tions

This reverts commit dc536fb4d3c8a01168de5d05e8562193dd68a88e.

steipete · 2026-04-26T04:14:48Z

Landed via rebase onto latest main.

Proof:

pnpm check:changed
pnpm check
pnpm test src/media-understanding/apply.test.ts src/media-understanding/format.test.ts src/media-understanding/runner.skip-tiny-audio.test.ts
CI completed on the merged PR head before merge; Silent/empty voice notes should inject a clear message instead of appearing as an API error #48944 closed by this PR.

Land commit: 360e12cff15bea26df74c4f78b579c3bbfeab3d1
Merge commit: 008e4ca81f7316f2df9feae234eb1e35bfe6ad21

Thanks @eulicesl!

* fix: add placeholder transcript for silent voice notes * fix: handle placeholder transcripts per skipped attachment * fix: preserve synthetic transcript attachment order * fix: scope synthetic audio merge to audio slice only, preserve cross-capability and prefer ordering Replace the global outputs.sort() with a targeted merge that: 1. Only sorts within the audio output slice (real + synthetic), preserving CAPABILITY_ORDER and per-capability attachments.prefer ordering for non-audio outputs. 2. Excludes synthetic placeholder indexes from audioAttachmentIndexes used by extractFileBlocks, so tiny audio-MIME files with text extensions can still be recovered via forcedTextMime. Adds mergeAudioOutputsPreservingAttachmentOrder helper. * fix: remove unused function and use toSorted() for oxlint compliance * fix(media-understanding): preserve selected audio order for synthetic placeholders - merge synthetic skipped-audio placeholders using audio decision order instead of raw attachmentIndex sorting, preserving attachments.prefer - insert synthetic-only audio outputs at the audio capability slot (before video) when no real audio outputs were produced * fix(media-understanding): use neutral too-small placeholder text Clarify that this synthetic transcript path is triggered by attachment size, not by a silence/no-speech detection result. * test(media-understanding): update too-small audio placeholder expectations * test(media-understanding): cover mixed too-small audio placeholder * test(media-understanding): cover too-small audio context * fix(tasks): preserve visible task title before internal context * Revert "fix(tasks): preserve visible task title before internal context" This reverts commit dc536fb4d3c8a01168de5d05e8562193dd68a88e. --------- Co-authored-by: Eulices Lopez <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added the size: S label Mar 17, 2026

chatgpt-codex-connector Bot reviewed Mar 17, 2026

View reviewed changes