Skip to content

Bug: OpenAI driver serializes image blocks as image_url without checking model vision capability (text-only models return HTTP 400) #6010

Description

@nevgenov

Symptom

A Telegram bot (bot-A) routes to an agent (agent-A) backed by a text-only model (DeepSeek deepseek-v4, supports_vision = false). When a user sends a photo, the bot replies:

Agent error: LLM driver error: Request failed:
{"error":{"message":"Failed to deserialize the JSON body into the target type:
messages[30]: unknown variant `image_url`, expected `text` at line 1 column 147537"}}

Column ~147537 is the inlined base64 of the downscaled image — i.e. the driver embedded the image as a data: URL into the chat-completions request sent to a model whose API only accepts text content parts.

This reproduces for any sidecar/channel bot whose default agent runs on a text-only OpenAI-compatible model. Vision-capable models are unaffected.

Root cause (read from current main)

crates/librefang-llm-drivers/src/drivers/openai.rs, build_request() (lines ~900-924) serializes both image block variants into image_url unconditionally — there is no supports_vision gate:

// crates/librefang-llm-drivers/src/drivers/openai.rs:900
ContentBlock::Image { media_type, data } => {
    parts.push(OaiContentPart::ImageUrl {
        image_url: OaiImageUrl {
            url: format!("data:{media_type};base64,{data}"),
        },
    });
}
ContentBlock::ImageFile { media_type, path } => {
    match tokio::task::block_in_place(|| std::fs::read(path)) {
        Ok(bytes) => {
            let data = base64::engine::general_purpose::STANDARD.encode(&bytes);
            parts.push(OaiContentPart::ImageUrl {
                image_url: OaiImageUrl {
                    url: format!("data:{media_type};base64,{data}"),
                },
            });
        }
        Err(e) => { warn!(path = %path, error = %e, "ImageFile missing, skipping"); }
    }
}

The only image-aware preprocessing is Moonshot-gated:

// crates/librefang-llm-drivers/src/drivers/openai.rs:1131
if self.is_moonshot() {
    self.preprocess_moonshot_files(&mut request).await?;
}

So for a non-Moonshot, non-vision model (DeepSeek, Mistral, etc.) the image block reaches the API verbatim as image_url, and the provider rejects the request with HTTP 400 unknown variant image_url, expected text.

CompletionRequest (crates/librefang-llm-driver/src/lib.rs:245) carries only model: String — no capability flag — so build_request has no per-request vision signal today; the driver relies on name heuristics (is_moonshot(), is_deepseek_*).

Why the inbound [media] description does not save it

The channel layer already downloads the photo to disk and calls the inbound image-description path, which prepends a textual description to the message. But it does not strip the ContentBlock::Image/ImageFile block — so even when a description is present, the original image block is still serialized to image_url and the 400 still fires.

Proposed fix

Generalize the existing Moonshot precedent (preprocess_moonshot_files, openai.rs:241, which already replaces eligible image/file blocks with a text marker) to all non-vision targets: when the target model is not vision-capable, replace ContentBlock::Image/ContentBlock::ImageFile with a text marker (e.g. the inbound [media] description, or a [image omitted: <media_type>] placeholder) instead of emitting OaiContentPart::ImageUrl.

Capability metadata for exactly this routing already exists from #4702 (LlmDriver capability metadata + capability-aware FallbackChain routing for vision/image_file) — plumbing a supports_vision signal into build_request (or gating the image arms behind it) would close the gap.

Minimal: in the ContentBlock::Image/ImageFile arms of build_request, when the model is not vision-capable, push a Text marker instead of ImageUrl.

Setup (for reference)

  • v2026.5.31-beta.16
  • Pattern: channel/sidecar bot whose default agent uses a text-only OpenAI-compatible model (supports_vision = false); user sends an inbound photo.
  • Inbound image description is enabled ([media] image_description = true), but the image block is still serialized → 400.

Happy to test a patch.

Refs: #4702 (capability metadata / fallback routing precedent)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiREST/WS endpoints and dashboardarea/runtimeAgent loop, LLM drivers, WASM sandboxarea/securitySecurity systems and auditinghas-prA pull request has been linked to this issue

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions