fix(channels): enrich coalesced mixed-media batches on the debounced path (#6348)#6351
Merged
Conversation
…path (#6348) Coalesced batches that mixed an image with other media delivered the non-image attachments (voice, audio, documents) to the agent as bare `[Voice message (Ns): url]` / `[File (name): url]` placeholders instead of their transcript or extracted text. Only images were pre-downloaded and enriched on the debounced (`message_coalesce_window_ms > 0`) path; audio/voice/document enrichment lived solely in the immediate `dispatch_message` path, so a photo+voice burst lost the voice transcription. This generalizes the image-only #6321 / #6323 parity fix to every media type by introducing two helpers that both inbound paths funnel through, so they cannot drift out of parity again. - `download_media_blocks` downloads any attachment (image / file / voice / audio / video) to disk and builds its blocks; it is I/O only, so it is safe on the debounce ingest loop, and returns `None` for non-media or a failed download so the caller falls back to the text placeholder. - `enrich_media` applies the deferred LLM enrichment (image description / audio transcription) in the dispatch task, off the ingest loop. The debounce ingest now pre-downloads all media (not just images) while each message's individual content is still available, and the flush task enriches every coalesced attachment via the same `enrich_media` the immediate path uses. Document extraction is already baked into `download_file_to_blocks`, so a coalesced PDF is enriched at download time regardless of path. The five per-type media arms in `dispatch_message` collapse into one `download_media_blocks` + `enrich_media` call, preserving their exact output (context headers, transcription inserted after the header, no description for image-typed documents).
houko
commented
Jun 29, 2026
houko
left a comment
Contributor
Author
There was a problem hiding this comment.
Style: multi-line comment blocks violate the project's one-line-max rule
CLAUDE.md (and the project system prompt): "Never write multi-paragraph docstrings or multi-line comment blocks — one short line max."
The following new items in bridge.rs violate this:
Multi-paragraph doc comments (need to be reduced to one line each):
DownloadedMediastruct — 4-line block including an empty///separatordownload_media_blocksfn — 5-line block with two///paragraphsenrich_mediafn — 4-line blockPendingMessage.mediafield — 3-line doc comment
Multi-line inline comment blocks (each should collapse to ≤1 line or be removed):
- The 2-line block before
return Some((pm.message, pm.media.into_iter().collect()));indrain() - The 2-line block before
let mut media: Vec<DownloadedMedia>indrain() - The 2-line block before
for m in media {inflush_debounced() - The 3-line block before
let media = download_media_blocks(...)in the debounce ingest loop - The 3-line block before
if let Some(media) = download_media_blocks(...)indispatch_message()
Comments explaining why (non-obvious invariants, issue references) are welcome, but must fit in one line. Please condense.
Generated by Claude Code
added 2 commits
June 29, 2026 12:08
Add `test_debouncer_mixed_media_collects_each_payload`, which drives the exact bug location — `MessageDebouncer::drain` — with an image coalesced with a voice note and asserts both `DownloadedMedia` payloads survive in arrival order (image → describe, voice → transcribe) and both placeholders remain in the merged text for sanitizer visibility. This covers the coalescing path directly, with no network, complementing the `enrich_media` unit tests. Also correct the `prepend_image_descriptions` doc comment: both inbound paths now reach it through `enrich_media`'s `DescribeImage` arm rather than calling it directly.
houko
enabled auto-merge (squash)
June 29, 2026 05:48
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6348.
Problem
On a channel bot with a coalesce window (
message_coalesce_window_ms > 0), when an inbound batch contained an image together with other media (e.g. a photo + a voice note within the debounce window), the non-image media reached the agent as a bare text placeholder —[Voice message (Ns): url],[File (name): url]— instead of its transcript / extracted text.The gap was narrow but real:
blocks == None→dispatch_messageblocks == Some→ debounced flushThis is the same class of divergence #6321 / #6323 fixed for images — but only images were made path-agnostic. The debounce ingest pre-downloaded only images, and the debounced flush only ran
prepend_image_descriptions; audio transcription and document extraction lived solely indispatch_message's per-type arms and were never reached for a mixed batch.Fix
Generalize the #6321 approach to all media types by funnelling both inbound paths through two shared helpers, so they cannot drift out of parity again:
download_media_blocks(handle, adapter, content) -> Option<DownloadedMedia>— downloads any attachment (image / file / voice / audio / video) to disk and builds its blocks (caption / context headers, theImageFileor[File: …] saved to …marker, and any document extraction already baked intodownload_file_to_blocks). It is I/O-bound only, so it is safe to run inline on the debounce ingest loop exactly as the original image-only pre-download was. It returnsNonefor non-media content and for a failed download, so the caller falls back to the text placeholder — matching the immediate path's fall-through.enrich_media(handle, media) -> Vec<ContentBlock>— applies the deferred, LLM-backed enrichment recorded on the payload (image description viaprepend_image_descriptions; audio transcription viamaybe_transcribe_inbound_audio, inserted right after the context header) in the dispatch task, off the ingest loop.Wiring:
enrich_mediathe immediate path uses, concatenates in arrival order, and prepends the coalesced text (which keeps media captions inside the sanitizer-checked text).dispatch_message's five per-type media arms collapse into onedownload_media_blocks+enrich_media+dispatch_with_blockscall, preserving their exact output: context headers, transcription inserted after the header (index 1), and — deliberately — no image description for image-typed documents (matching the old File arm).Net effect: a mixed photo+voice+pdf burst reaches the agent as one turn with every attachment turned into text, identically whether or not it was coalesced — which also benefits text-only models.
Verification
cargo check -p librefang-channels --lib— clean (no warnings).enrich_mediaunit tests next to the existingmaybe_describe_inbound_image/maybe_transcribe_inbound_audiofakes:enrich_media_describe_image_describes_each_imagefile—DescribeImageinserts the description before theImageFile.enrich_media_none_leaves_blocks_and_skips_vision—Nonereturns blocks verbatim with no kernel call (an image-typed document is not described).enrich_media_transcribe_audio_inserts_after_header—TranscribeAudioinserts[Transcription: …]between the[Voice message …]header and the saved-path block (same order the immediate path'stranscription_block_lands_between_header_and_file_pathtest pins).enrich_media_transcribe_audio_noop_when_disabled—Ok(None)(feature off) leaves the [header, saved-path] pair unchanged.Vec<DownloadedMedia>return shape.Live LLM round-trip verification (a real coalesced photo+voice burst against a running daemon) is left for a maintainer per the repo's human-only integration-testing policy.