Symptom
A Telegram bot (bot-A) routes to an agent (agent-A) backed by a text-only model (DeepSeek deepseek-v4, supports_vision = false). When a user sends a photo, the bot replies:
Agent error: LLM driver error: Request failed:
{"error":{"message":"Failed to deserialize the JSON body into the target type:
messages[30]: unknown variant `image_url`, expected `text` at line 1 column 147537"}}
Column ~147537 is the inlined base64 of the downscaled image — i.e. the driver embedded the image as a data: URL into the chat-completions request sent to a model whose API only accepts text content parts.
This reproduces for any sidecar/channel bot whose default agent runs on a text-only OpenAI-compatible model. Vision-capable models are unaffected.
Root cause (read from current main)
crates/librefang-llm-drivers/src/drivers/openai.rs, build_request() (lines ~900-924) serializes both image block variants into image_url unconditionally — there is no supports_vision gate:
// crates/librefang-llm-drivers/src/drivers/openai.rs:900
ContentBlock::Image { media_type, data } => {
parts.push(OaiContentPart::ImageUrl {
image_url: OaiImageUrl {
url: format!("data:{media_type};base64,{data}"),
},
});
}
ContentBlock::ImageFile { media_type, path } => {
match tokio::task::block_in_place(|| std::fs::read(path)) {
Ok(bytes) => {
let data = base64::engine::general_purpose::STANDARD.encode(&bytes);
parts.push(OaiContentPart::ImageUrl {
image_url: OaiImageUrl {
url: format!("data:{media_type};base64,{data}"),
},
});
}
Err(e) => { warn!(path = %path, error = %e, "ImageFile missing, skipping"); }
}
}
The only image-aware preprocessing is Moonshot-gated:
// crates/librefang-llm-drivers/src/drivers/openai.rs:1131
if self.is_moonshot() {
self.preprocess_moonshot_files(&mut request).await?;
}
So for a non-Moonshot, non-vision model (DeepSeek, Mistral, etc.) the image block reaches the API verbatim as image_url, and the provider rejects the request with HTTP 400 unknown variant image_url, expected text.
CompletionRequest (crates/librefang-llm-driver/src/lib.rs:245) carries only model: String — no capability flag — so build_request has no per-request vision signal today; the driver relies on name heuristics (is_moonshot(), is_deepseek_*).
Why the inbound [media] description does not save it
The channel layer already downloads the photo to disk and calls the inbound image-description path, which prepends a textual description to the message. But it does not strip the ContentBlock::Image/ImageFile block — so even when a description is present, the original image block is still serialized to image_url and the 400 still fires.
Proposed fix
Generalize the existing Moonshot precedent (preprocess_moonshot_files, openai.rs:241, which already replaces eligible image/file blocks with a text marker) to all non-vision targets: when the target model is not vision-capable, replace ContentBlock::Image/ContentBlock::ImageFile with a text marker (e.g. the inbound [media] description, or a [image omitted: <media_type>] placeholder) instead of emitting OaiContentPart::ImageUrl.
Capability metadata for exactly this routing already exists from #4702 (LlmDriver capability metadata + capability-aware FallbackChain routing for vision/image_file) — plumbing a supports_vision signal into build_request (or gating the image arms behind it) would close the gap.
Minimal: in the ContentBlock::Image/ImageFile arms of build_request, when the model is not vision-capable, push a Text marker instead of ImageUrl.
Setup (for reference)
- v2026.5.31-beta.16
- Pattern: channel/sidecar bot whose default agent uses a text-only OpenAI-compatible model (
supports_vision = false); user sends an inbound photo.
- Inbound image description is enabled (
[media] image_description = true), but the image block is still serialized → 400.
Happy to test a patch.
Refs: #4702 (capability metadata / fallback routing precedent)
Symptom
A Telegram bot (
bot-A) routes to an agent (agent-A) backed by a text-only model (DeepSeekdeepseek-v4,supports_vision = false). When a user sends a photo, the bot replies:Column ~147537 is the inlined base64 of the downscaled image — i.e. the driver embedded the image as a
data:URL into the chat-completions request sent to a model whose API only accepts text content parts.This reproduces for any sidecar/channel bot whose default agent runs on a text-only OpenAI-compatible model. Vision-capable models are unaffected.
Root cause (read from current
main)crates/librefang-llm-drivers/src/drivers/openai.rs,build_request()(lines ~900-924) serializes both image block variants intoimage_urlunconditionally — there is nosupports_visiongate:The only image-aware preprocessing is Moonshot-gated:
So for a non-Moonshot, non-vision model (DeepSeek, Mistral, etc.) the image block reaches the API verbatim as
image_url, and the provider rejects the request with HTTP 400unknown variant image_url, expected text.CompletionRequest(crates/librefang-llm-driver/src/lib.rs:245) carries onlymodel: String— no capability flag — sobuild_requesthas no per-request vision signal today; the driver relies on name heuristics (is_moonshot(),is_deepseek_*).Why the inbound
[media]description does not save itThe channel layer already downloads the photo to disk and calls the inbound image-description path, which prepends a textual description to the message. But it does not strip the
ContentBlock::Image/ImageFileblock — so even when a description is present, the original image block is still serialized toimage_urland the 400 still fires.Proposed fix
Generalize the existing Moonshot precedent (
preprocess_moonshot_files,openai.rs:241, which already replaces eligible image/file blocks with a text marker) to all non-vision targets: when the target model is not vision-capable, replaceContentBlock::Image/ContentBlock::ImageFilewith a text marker (e.g. the inbound[media]description, or a[image omitted: <media_type>]placeholder) instead of emittingOaiContentPart::ImageUrl.Capability metadata for exactly this routing already exists from #4702 (LlmDriver capability metadata + capability-aware FallbackChain routing for vision/image_file) — plumbing a
supports_visionsignal intobuild_request(or gating the image arms behind it) would close the gap.Minimal: in the
ContentBlock::Image/ImageFilearms ofbuild_request, when the model is not vision-capable, push aTextmarker instead ofImageUrl.Setup (for reference)
supports_vision = false); user sends an inbound photo.[media] image_description = true), but the image block is still serialized → 400.Happy to test a patch.
Refs: #4702 (capability metadata / fallback routing precedent)