Skip to content

fix(channels): enrich coalesced mixed-media batches on the debounced path (#6348)#6351

Merged
houko merged 3 commits into
mainfrom
fix/coalesce-media-enrich
Jun 29, 2026
Merged

fix(channels): enrich coalesced mixed-media batches on the debounced path (#6348)#6351
houko merged 3 commits into
mainfrom
fix/coalesce-media-enrich

Conversation

@houko

@houko houko commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Fixes #6348.

Problem

On a channel bot with a coalesce window (message_coalesce_window_ms > 0), when an inbound batch contained an image together with other media (e.g. a photo + a voice note within the debounce window), the non-image media reached the agent as a bare text placeholder — [Voice message (Ns): url], [File (name): url] — instead of its transcript / extracted text.

The gap was narrow but real:

Case Path Pre-fix result
Voice/doc alone, or in an image-free batch blocks == Nonedispatch_message transcribed / enriched ✅
Voice/doc coalesced with an image blocks == Some → debounced flush placeholder text, not transcribed/enriched ❌

This is the same class of divergence #6321 / #6323 fixed for images — but only images were made path-agnostic. The debounce ingest pre-downloaded only images, and the debounced flush only ran prepend_image_descriptions; audio transcription and document extraction lived solely in dispatch_message's per-type arms and were never reached for a mixed batch.

Fix

Generalize the #6321 approach to all media types by funnelling both inbound paths through two shared helpers, so they cannot drift out of parity again:

  • download_media_blocks(handle, adapter, content) -> Option<DownloadedMedia> — downloads any attachment (image / file / voice / audio / video) to disk and builds its blocks (caption / context headers, the ImageFile or [File: …] saved to … marker, and any document extraction already baked into download_file_to_blocks). It is I/O-bound only, so it is safe to run inline on the debounce ingest loop exactly as the original image-only pre-download was. It returns None for non-media content and for a failed download, so the caller falls back to the text placeholder — matching the immediate path's fall-through.
  • enrich_media(handle, media) -> Vec<ContentBlock> — applies the deferred, LLM-backed enrichment recorded on the payload (image description via prepend_image_descriptions; audio transcription via maybe_transcribe_inbound_audio, inserted right after the context header) in the dispatch task, off the ingest loop.

Wiring:

  • The debounce ingest now pre-downloads every media type (not just images) while the message's individual content is still available — after coalescing the debouncer only retains a merged text placeholder.
  • The debounced flush enriches each coalesced attachment with the same enrich_media the immediate path uses, concatenates in arrival order, and prepends the coalesced text (which keeps media captions inside the sanitizer-checked text).
  • dispatch_message's five per-type media arms collapse into one download_media_blocks + enrich_media + dispatch_with_blocks call, preserving their exact output: context headers, transcription inserted after the header (index 1), and — deliberately — no image description for image-typed documents (matching the old File arm).

Net effect: a mixed photo+voice+pdf burst reaches the agent as one turn with every attachment turned into text, identically whether or not it was coalesced — which also benefits text-only models.

Verification

  • cargo check -p librefang-channels --lib — clean (no warnings).
  • New enrich_media unit tests next to the existing maybe_describe_inbound_image / maybe_transcribe_inbound_audio fakes:
    • enrich_media_describe_image_describes_each_imagefileDescribeImage inserts the description before the ImageFile.
    • enrich_media_none_leaves_blocks_and_skips_visionNone returns blocks verbatim with no kernel call (an image-typed document is not described).
    • enrich_media_transcribe_audio_inserts_after_headerTranscribeAudio inserts [Transcription: …] between the [Voice message …] header and the saved-path block (same order the immediate path's transcription_block_lands_between_header_and_file_path test pins).
    • enrich_media_transcribe_audio_noop_when_disabledOk(None) (feature off) leaves the [header, saved-path] pair unchanged.
  • Existing debouncer drain tests updated for the new Vec<DownloadedMedia> return shape.
  • clippy + full test suite (including the integration shards) run in CI.

Live LLM round-trip verification (a real coalesced photo+voice burst against a running daemon) is left for a maintainer per the repo's human-only integration-testing policy.

…path (#6348)

Coalesced batches that mixed an image with other media delivered the non-image attachments (voice, audio, documents) to the agent as bare `[Voice message (Ns): url]` / `[File (name): url]` placeholders instead of their transcript or extracted text.
Only images were pre-downloaded and enriched on the debounced (`message_coalesce_window_ms > 0`) path; audio/voice/document enrichment lived solely in the immediate `dispatch_message` path, so a photo+voice burst lost the voice transcription.

This generalizes the image-only #6321 / #6323 parity fix to every media type by introducing two helpers that both inbound paths funnel through, so they cannot drift out of parity again.

- `download_media_blocks` downloads any attachment (image / file / voice / audio / video) to disk and builds its blocks; it is I/O only, so it is safe on the debounce ingest loop, and returns `None` for non-media or a failed download so the caller falls back to the text placeholder.
- `enrich_media` applies the deferred LLM enrichment (image description / audio transcription) in the dispatch task, off the ingest loop.

The debounce ingest now pre-downloads all media (not just images) while each message's individual content is still available, and the flush task enriches every coalesced attachment via the same `enrich_media` the immediate path uses.
Document extraction is already baked into `download_file_to_blocks`, so a coalesced PDF is enriched at download time regardless of path.

The five per-type media arms in `dispatch_message` collapse into one `download_media_blocks` + `enrich_media` call, preserving their exact output (context headers, transcription inserted after the header, no description for image-typed documents).
@github-actions github-actions Bot added area/docs Documentation and guides area/channels Messaging channel adapters has-conflicts PR has merge conflicts that need resolution size/L 250-999 lines changed labels Jun 29, 2026

@houko houko left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style: multi-line comment blocks violate the project's one-line-max rule

CLAUDE.md (and the project system prompt): "Never write multi-paragraph docstrings or multi-line comment blocks — one short line max."

The following new items in bridge.rs violate this:

Multi-paragraph doc comments (need to be reduced to one line each):

  • DownloadedMedia struct — 4-line block including an empty /// separator
  • download_media_blocks fn — 5-line block with two /// paragraphs
  • enrich_media fn — 4-line block
  • PendingMessage.media field — 3-line doc comment

Multi-line inline comment blocks (each should collapse to ≤1 line or be removed):

  • The 2-line block before return Some((pm.message, pm.media.into_iter().collect())); in drain()
  • The 2-line block before let mut media: Vec<DownloadedMedia> in drain()
  • The 2-line block before for m in media { in flush_debounced()
  • The 3-line block before let media = download_media_blocks(...) in the debounce ingest loop
  • The 3-line block before if let Some(media) = download_media_blocks(...) in dispatch_message()

Comments explaining why (non-obvious invariants, issue references) are welcome, but must fit in one line. Please condense.


Generated by Claude Code

Evan added 2 commits June 29, 2026 12:08
Add `test_debouncer_mixed_media_collects_each_payload`, which drives the exact bug location — `MessageDebouncer::drain` — with an image coalesced with a voice note and asserts both `DownloadedMedia` payloads survive in arrival order (image → describe, voice → transcribe) and both placeholders remain in the merged text for sanitizer visibility.
This covers the coalescing path directly, with no network, complementing the `enrich_media` unit tests.

Also correct the `prepend_image_descriptions` doc comment: both inbound paths now reach it through `enrich_media`'s `DescribeImage` arm rather than calling it directly.
@github-actions github-actions Bot added ready-for-review PR is ready for maintainer review and removed has-conflicts PR has merge conflicts that need resolution labels Jun 29, 2026
@houko
houko enabled auto-merge (squash) June 29, 2026 05:48
@houko
houko merged commit e73b384 into main Jun 29, 2026
31 checks passed
@houko
houko deleted the fix/coalesce-media-enrich branch June 29, 2026 05:58
@houko houko mentioned this pull request Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/channels Messaging channel adapters area/docs Documentation and guides ready-for-review PR is ready for maintainer review size/L 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: coalesced mixed-media batches lose non-image enrichment (audio transcript / doc extract) on the debounced path

1 participant