fix(channels): describe inbound images on the debounced channel path (#6321)#6323
Merged
Conversation
…6321) The image-description step (`maybe_describe_inbound_image`, gated by `[media] image_description`) was only reachable from the immediate dispatch path `dispatch_message`; the debounced path (`message_coalesce_window_ms > 0`) assembled its image blocks in `flush_debounced` and called `dispatch_with_blocks` directly, so it never described inbound photos. On a text-only model (`supports_vision = false`) the un-described image is then redacted to the `[image omitted: …]` placeholder by the #6010 vision gate, so the agent silently loses all vision — there is no HTTP 400 or crash, just "the bot ignores photos" on coalesced bots. Extract the per-image describe-and-prepend step into one shared helper, `prepend_image_descriptions`, and call it from both the immediate and debounced paths so the two cannot drift apart again. Because the debounced path coalesces several messages it can carry multiple `ImageFile` blocks, so the helper describes every image — inserting each description immediately before its block — not just the first.
Three comment blocks added in this PR wrapped mid-sentence at ~80 chars rather than at sentence boundaries, violating the repo's prose-wrapping rule (CLAUDE.md: "one sentence = one line, regardless of length"). Reformatted the /// doc-comment on `prepend_image_descriptions` and the // blocks in `flush_debounced` and `dispatch_message` so every sentence is its own line. No code logic changed.
houko
commented
Jun 25, 2026
houko
left a comment
Contributor
Author
There was a problem hiding this comment.
Code is correct and well-tested (three new unit tests cover the multi-image, disabled, and text-only cases). One comment-style issue to fix before merge — see inline note.
Generated by Claude Code
Contributor
Author
There was a problem hiding this comment.
CLAUDE.md: "Don't write multi-paragraph docstrings or multi-line comment blocks — one short line max."
Three new comment blocks added in this PR violate that rule:
- The
///doc onprepend_image_descriptionsspans nine lines across four paragraphs. - The
//block at theflush_debouncedcall-site is three lines. - The
//block replacing the old comment indispatch_messageis three lines.
The WHY is non-obvious enough to warrant a comment on each — just collapse to one line:
/// Prepend a vision-description text block before every `ImageFile` in `blocks` so both inbound paths describe images identically; self-gates on `[media] image_description` (refs #6321).
// Parity with `dispatch_message`: describe inbound images for text-only models via the shared helper (refs #6321).// Shared helper keeps both paths in parity (refs #6321).Generated by Claude Code
This was referenced Jun 26, 2026
houko
added a commit
that referenced
this pull request
Jun 29, 2026
…path (#6348) (#6351) * fix(channels): enrich coalesced mixed-media batches on the debounced path (#6348) Coalesced batches that mixed an image with other media delivered the non-image attachments (voice, audio, documents) to the agent as bare `[Voice message (Ns): url]` / `[File (name): url]` placeholders instead of their transcript or extracted text. Only images were pre-downloaded and enriched on the debounced (`message_coalesce_window_ms > 0`) path; audio/voice/document enrichment lived solely in the immediate `dispatch_message` path, so a photo+voice burst lost the voice transcription. This generalizes the image-only #6321 / #6323 parity fix to every media type by introducing two helpers that both inbound paths funnel through, so they cannot drift out of parity again. - `download_media_blocks` downloads any attachment (image / file / voice / audio / video) to disk and builds its blocks; it is I/O only, so it is safe on the debounce ingest loop, and returns `None` for non-media or a failed download so the caller falls back to the text placeholder. - `enrich_media` applies the deferred LLM enrichment (image description / audio transcription) in the dispatch task, off the ingest loop. The debounce ingest now pre-downloads all media (not just images) while each message's individual content is still available, and the flush task enriches every coalesced attachment via the same `enrich_media` the immediate path uses. Document extraction is already baked into `download_file_to_blocks`, so a coalesced PDF is enriched at download time regardless of path. The five per-type media arms in `dispatch_message` collapse into one `download_media_blocks` + `enrich_media` call, preserving their exact output (context headers, transcription inserted after the header, no description for image-typed documents). * test(channels): drain coalescing regression for mixed media (#6348) Add `test_debouncer_mixed_media_collects_each_payload`, which drives the exact bug location — `MessageDebouncer::drain` — with an image coalesced with a voice note and asserts both `DownloadedMedia` payloads survive in arrival order (image → describe, voice → transcribe) and both placeholders remain in the merged text for sanitizer visibility. This covers the coalescing path directly, with no network, complementing the `enrich_media` unit tests. Also correct the `prepend_image_descriptions` doc comment: both inbound paths now reach it through `enrich_media`'s `DescribeImage` arm rather than calling it directly. --------- Co-authored-by: Evan <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #6321. Inbound channel images were only described on the immediate dispatch path; the debounced (message-coalesce) path never ran the description step, so text-only models lost vision on coalesced bots.
The kernel's image-description step (
maybe_describe_inbound_image, gated by[media] image_description) was reachable only fromdispatch_message(themessage_coalesce_window_ms == 0fast path). The debounced path (message_coalesce_window_ms > 0) assembles its image blocks inflush_debouncedand calls the shareddispatch_with_blockstail directly, which contains no description call. So[media] image_descriptionwas only ever consulted when coalescing was off.On a text-only model the un-described
ImageFileblock is then redacted to the[image omitted: …]placeholder by the #6010 vision gate, so the failure is silent — no HTTP 400, no crash, the agent simply replies that it cannot see the image. Vision-capable models were unaffected (they read the raw bytes either way), which is why this only surfaced on text-only + coalesce setups.Change
prepend_image_descriptions(handle, blocks), placed next tomaybe_describe_inbound_image.dispatch_message(immediate) andflush_debounced(debounced) right beforedispatch_with_blocks, so the description step lives in one place and the two paths cannot drift out of parity again — the root cause was exactly that divergence.ImageFileblock, not just the first, inserting each description immediately before its image. This matters becauseMessageDebouncer::draincoalesces the image blocks of several buffered messages into one list (all_blocks.extend(...)), so a debounced batch can legitimately carry multiple images.The step self-gates inside
describe_inbound_image([media] image_description), so it remains a cheap no-op when the operator has not opted in, and a failed/timed-out description still degrades to the existing opaque[Image description unavailable]note rather than dropping the image.Behavior note (intentional)
On the immediate path the description used to be prepended at index 0 (before the caption); it is now inserted immediately before the image block (after the caption), matching the debounced path. The description is still adjacent to and precedes the image, so model behavior is unchanged; the ordering is now consistent across both paths. No test asserted the old absolute-front position.
Verification
cargo check -p librefang-channels --lib— cleancargo clippy -p librefang-channels --all-targets -- -D warnings— zero warningscargo test -p librefang-channels --lib— 510 passed, 0 failedcargo fmt -p librefang-channels -- --check— cleanNew regression tests in
bridge::tests::inbound_image_description(reuse the existingDescribeHandlefake):prepend_describes_every_imagefile_before_its_block— multi-image list gets a description before each image; both images hit the vision path.prepend_is_noop_when_description_disabled—Ok(None)(feature off) leaves blocks unchanged.prepend_leaves_non_image_blocks_untouched— a text-only list triggers no vision call.The existing
maybe_describe_inbound_imagecases (success / disabled / provider-failure-doesn't-leak / timeout / non-image-mime) continue to pass.Out of scope
Live LLM round-trip verification (real Groq vision call against a coalesced bot) requires a running daemon and is left for a maintainer per the repo's human-only integration-testing policy.