fix(runtime,kernel): redact images for text-only models via catalog supports_vision (#6010)#6013
Conversation
cdd8be2 to
dd59049
Compare
github-actions
Bot
added
area/runtime
houko
changed the title
run_agent_loop_streaming built CompletionRequest using the raw message list, so a text-only model (supports_vision = false) still received image content blocks and returned HTTP 400. Apply the same catalog-driven supports_vision gate introduced for the non-streaming path: look up supports_vision_for from the kernel handle, redact image blocks via redact_images_for_text_only when false. Make redact_images_for_text_only pub(super) so run_streaming.rs can call it.
houko
left a comment
houko
left a comment
There was a problem hiding this comment.
Streaming path was unguarded — pushed a fix.
run_agent_loop_streaming built CompletionRequest from messages.clone() directly, skipping the supports_vision_for check added to the non-streaming path.
Any production deployment using the WebUI or channel bots (which run the streaming loop) would still receive HTTP 400 on image content with a text-only model.
Fix applied: redact_images_for_text_only is now pub(super) and the streaming loop mirrors the non-streaming vision gate before constructing CompletionRequest.
Advisory: lookup_supports_vision inlines what effective_capabilities_for already provides
In crates/librefang-kernel/src/kernel/handles/catalog_query.rs, the implementation:
catalog.find_model(model).map(|m| catalog.effective_capabilities(m).supports_vision).unwrap_or(true)duplicates the two-step logic that catalog.effective_capabilities_for(model) already encapsulates (that method exists at model_catalog.rs:735).
Using it would reduce this to a one-liner:
catalog.effective_capabilities_for(model).map(|c| c.supports_vision).unwrap_or(true)Not a bug, but inconsistency with the existing API surface means a future change to effective_capabilities_for's logic would need to be mirrored here manually.
Advisory: supports_vision_for is re-evaluated inside the iteration loop
api_model is constant across loop iterations, but supports_vision_for does a fresh ArcSwap::load() on each call.
A catalog hot-reload mid-turn could flip vision support between iterations: early iterations send images, later ones redact them, and the session history becomes inconsistent.
The reasoning_echo_policy lookup has the same pattern, so this is not unique to this PR, but it is worth documenting the "snapshot per iteration" semantics or hoisting both lookups before the loop for consistency.
Generated by Claude Code
2 participants
Problem
A channel/sidecar bot whose default agent runs a text-only model (e.g.
DeepSeek
deepseek-v4,supports_vision = false) breaks the moment a usersends a photo. The OpenAI-compatible API rejects image content parts with
HTTP 400:
The channel layer downloads the image and prepends a textual
[media]description, but it does not strip the
ContentBlock::Image/ImageFileblock, so the image still reaches the driver, is inlined as a
data:URL, and400s the whole turn. This affects every entry path with a text-only default
agent (WebUI, channels, sidecars, triggers).
Approach (authoritative, catalog-driven)
The previous revision of this PR used a driver-local name deny-list in
openai.rs. That was rejected by the maintainer as non-authoritative. Thisrevision moves the gate to the model catalog and redacts upstream of the
driver, so it covers all entry paths and all OpenAI-compatible drivers,
respects user capability overrides, and never hard-codes provider names.
LibreFang's authoritative vision flag is the model catalog's
effective_capabilities().supports_vision, which already honours useroverrides (#4745). This is surfaced to request-build time via the same
CatalogQuerytrait that #4842 used to threadreasoning_echo_policy.Changes
CatalogQuery::supports_vision_for(model) -> bool— new trait methodmirroring
reasoning_echo_policy_for, with a default body returningtrue(fail open) so non-overriding mocks/stubs keep sending images unchanged.
crates/librefang-kernel-handle/src/catalog_query.rsKernel override — resolves the flag from the catalog, honouring user
overrides via
effective_capabilities, fails open on a catalog miss(
unwrap_or(true)), using the same catalog accessor aslookup_reasoning_echo_policyand the same call shape asws.rs.crates/librefang-kernel/src/kernel/handles/catalog_query.rsAgent-loop gate — at the request-build site in
run_agent_loop, look upsupports_visionvia the kernel handle (samekernel.as_ref().map(|k| …)pattern as the adjacent
reasoning_echo_policylookup). When!supports_vision, redact image blocks to a text placeholder via a smallpure helper
redact_images_for_text_only(messages, model)beforebuilding the request. The vision path is byte-identical to before.
crates/librefang-runtime/src/agent_loop/mod.rsopenai.rsreverted tomain— the driver again serializes imagesunconditionally; the gate now lives upstream.
Why this is correct
(
effective_capabilities().supports_vision), not a name heuristic.effective_capabilitiesapplies theper-model override, so an operator who force-enables vision on a
mis-declared model keeps sending images.
inbound surface (WebUI, channels, sidecars, triggers) is protected, not just
the OpenAI driver / a known family list.
unchanged, so vision and unknown models are never degraded.
Tests
crates/librefang-kernel-handle/src/tests.rs—catalog_query_supports_vision_defaults_to_true: a stub that doesn'toverride the method returns
true(mirrors the existingcatalog_query_default_returns_nonetest).crates/librefang-runtime/src/agent_loop/tests/utilities.rs—redact_images_for_text_only_replaces_image_and_imagefile_blocks: bothImageandImageFileblocks become text placeholders that mention themodel; text/other blocks and block counts are preserved.
redact_images_for_text_only_is_noop_without_images: messages without imageblocks pass through structurally unchanged.
Verification
Local compilation/tests were not possible in this environment (no native
cargo/rustc, no reliable Docker). The change follows the existingreasoning_echo_policy_forprecedent andws.rscatalog-call shape, and isleft for CI to compile and run as the gate.
Closes #6010