Skip to content

fix(channels): describe inbound images on the debounced channel path (#6321)#6323

Merged
houko merged 2 commits into
mainfrom
fix-debounce-image-description
Jun 26, 2026
Merged

fix(channels): describe inbound images on the debounced channel path (#6321)#6323
houko merged 2 commits into
mainfrom
fix-debounce-image-description

Conversation

@houko

@houko houko commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #6321. Inbound channel images were only described on the immediate dispatch path; the debounced (message-coalesce) path never ran the description step, so text-only models lost vision on coalesced bots.

The kernel's image-description step (maybe_describe_inbound_image, gated by [media] image_description) was reachable only from dispatch_message (the message_coalesce_window_ms == 0 fast path). The debounced path (message_coalesce_window_ms > 0) assembles its image blocks in flush_debounced and calls the shared dispatch_with_blocks tail directly, which contains no description call. So [media] image_description was only ever consulted when coalescing was off.

On a text-only model the un-described ImageFile block is then redacted to the [image omitted: …] placeholder by the #6010 vision gate, so the failure is silent — no HTTP 400, no crash, the agent simply replies that it cannot see the image. Vision-capable models were unaffected (they read the raw bytes either way), which is why this only surfaced on text-only + coalesce setups.

Change

  • Extract the per-image describe-and-prepend logic into one shared helper, prepend_image_descriptions(handle, blocks), placed next to maybe_describe_inbound_image.
  • Call it from both dispatch_message (immediate) and flush_debounced (debounced) right before dispatch_with_blocks, so the description step lives in one place and the two paths cannot drift out of parity again — the root cause was exactly that divergence.
  • The helper describes every ImageFile block, not just the first, inserting each description immediately before its image. This matters because MessageDebouncer::drain coalesces the image blocks of several buffered messages into one list (all_blocks.extend(...)), so a debounced batch can legitimately carry multiple images.

The step self-gates inside describe_inbound_image ([media] image_description), so it remains a cheap no-op when the operator has not opted in, and a failed/timed-out description still degrades to the existing opaque [Image description unavailable] note rather than dropping the image.

Behavior note (intentional)

On the immediate path the description used to be prepended at index 0 (before the caption); it is now inserted immediately before the image block (after the caption), matching the debounced path. The description is still adjacent to and precedes the image, so model behavior is unchanged; the ordering is now consistent across both paths. No test asserted the old absolute-front position.

Verification

  • cargo check -p librefang-channels --lib — clean
  • cargo clippy -p librefang-channels --all-targets -- -D warnings — zero warnings
  • cargo test -p librefang-channels --lib510 passed, 0 failed
  • cargo fmt -p librefang-channels -- --check — clean

New regression tests in bridge::tests::inbound_image_description (reuse the existing DescribeHandle fake):

  • prepend_describes_every_imagefile_before_its_block — multi-image list gets a description before each image; both images hit the vision path.
  • prepend_is_noop_when_description_disabledOk(None) (feature off) leaves blocks unchanged.
  • prepend_leaves_non_image_blocks_untouched — a text-only list triggers no vision call.

The existing maybe_describe_inbound_image cases (success / disabled / provider-failure-doesn't-leak / timeout / non-image-mime) continue to pass.

Out of scope

Live LLM round-trip verification (real Groq vision call against a coalesced bot) requires a running daemon and is left for a maintainer per the repo's human-only integration-testing policy.

…6321)

The image-description step (`maybe_describe_inbound_image`, gated by `[media] image_description`) was only reachable from the immediate dispatch path `dispatch_message`; the debounced path (`message_coalesce_window_ms > 0`) assembled its image blocks in `flush_debounced` and called `dispatch_with_blocks` directly, so it never described inbound photos.

On a text-only model (`supports_vision = false`) the un-described image is then redacted to the `[image omitted: …]` placeholder by the #6010 vision gate, so the agent silently loses all vision — there is no HTTP 400 or crash, just "the bot ignores photos" on coalesced bots.

Extract the per-image describe-and-prepend step into one shared helper, `prepend_image_descriptions`, and call it from both the immediate and debounced paths so the two cannot drift apart again.

Because the debounced path coalesces several messages it can carry multiple `ImageFile` blocks, so the helper describes every image — inserting each description immediately before its block — not just the first.
@github-actions github-actions Bot added size/M 50-249 lines changed area/channels Messaging channel adapters labels Jun 25, 2026
Three comment blocks added in this PR wrapped mid-sentence at ~80 chars
rather than at sentence boundaries, violating the repo's prose-wrapping
rule (CLAUDE.md: "one sentence = one line, regardless of length").
Reformatted the /// doc-comment on `prepend_image_descriptions` and the
// blocks in `flush_debounced` and `dispatch_message` so every sentence
is its own line.
No code logic changed.

@houko houko left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code is correct and well-tested (three new unit tests cover the multi-image, disabled, and text-only cases). One comment-style issue to fix before merge — see inline note.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLAUDE.md: "Don't write multi-paragraph docstrings or multi-line comment blocks — one short line max."

Three new comment blocks added in this PR violate that rule:

  1. The /// doc on prepend_image_descriptions spans nine lines across four paragraphs.
  2. The // block at the flush_debounced call-site is three lines.
  3. The // block replacing the old comment in dispatch_message is three lines.

The WHY is non-obvious enough to warrant a comment on each — just collapse to one line:

/// Prepend a vision-description text block before every `ImageFile` in `blocks` so both inbound paths describe images identically; self-gates on `[media] image_description` (refs #6321).
// Parity with `dispatch_message`: describe inbound images for text-only models via the shared helper (refs #6321).
// Shared helper keeps both paths in parity (refs #6321).

Generated by Claude Code

@houko
houko merged commit 43db218 into main Jun 26, 2026
32 checks passed
@houko
houko deleted the fix-debounce-image-description branch June 26, 2026 05:28
houko added a commit that referenced this pull request Jun 29, 2026
…path (#6348) (#6351)

* fix(channels): enrich coalesced mixed-media batches on the debounced path (#6348)

Coalesced batches that mixed an image with other media delivered the non-image attachments (voice, audio, documents) to the agent as bare `[Voice message (Ns): url]` / `[File (name): url]` placeholders instead of their transcript or extracted text.
Only images were pre-downloaded and enriched on the debounced (`message_coalesce_window_ms > 0`) path; audio/voice/document enrichment lived solely in the immediate `dispatch_message` path, so a photo+voice burst lost the voice transcription.

This generalizes the image-only #6321 / #6323 parity fix to every media type by introducing two helpers that both inbound paths funnel through, so they cannot drift out of parity again.

- `download_media_blocks` downloads any attachment (image / file / voice / audio / video) to disk and builds its blocks; it is I/O only, so it is safe on the debounce ingest loop, and returns `None` for non-media or a failed download so the caller falls back to the text placeholder.
- `enrich_media` applies the deferred LLM enrichment (image description / audio transcription) in the dispatch task, off the ingest loop.

The debounce ingest now pre-downloads all media (not just images) while each message's individual content is still available, and the flush task enriches every coalesced attachment via the same `enrich_media` the immediate path uses.
Document extraction is already baked into `download_file_to_blocks`, so a coalesced PDF is enriched at download time regardless of path.

The five per-type media arms in `dispatch_message` collapse into one `download_media_blocks` + `enrich_media` call, preserving their exact output (context headers, transcription inserted after the header, no description for image-typed documents).

* test(channels): drain coalescing regression for mixed media (#6348)

Add `test_debouncer_mixed_media_collects_each_payload`, which drives the exact bug location — `MessageDebouncer::drain` — with an image coalesced with a voice note and asserts both `DownloadedMedia` payloads survive in arrival order (image → describe, voice → transcribe) and both placeholders remain in the merged text for sanitizer visibility.
This covers the coalescing path directly, with no network, complementing the `enrich_media` unit tests.

Also correct the `prepend_image_descriptions` doc comment: both inbound paths now reach it through `enrich_media`'s `DescribeImage` arm rather than calling it directly.

---------

Co-authored-by: Evan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/channels Messaging channel adapters size/M 50-249 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: debounced channel path skips inbound image description ([media] image_description), text-only models lose vision

2 participants