Skip to content

feat(core): describe images with a vision model when the session model cannot see them - #62534

Closed
SOUMITRO-SAHA wants to merge 1 commit into
anomalyco:devfrom
SOUMITRO-SAHA:image-fallback
Closed

SOUMITRO-SAHA wants to merge 1 commit into
anomalyco:devfrom
SOUMITRO-SAHA:image-fallback

Conversation

@SOUMITRO-SAHA

Copy link
Copy Markdown
Contributor

Issue for this PR

Ports #26164 to V2. That PR (and its issue #26160) were closed as V1-only with the ask to port the feature to V2.

Type of change

  • New feature

What does this PR do?

When the session model cannot see images, V2 currently replaces every image with ERROR: Cannot read "x.png" (this model does not support image input). Inform the user. (via unsupportedParts in SessionModelRequest). This PR instead describes the image once with a vision-capable model and substitutes that description, so the conversation can continue on a text-only model.

  • New SessionImageDescribe service runs inside SessionModelRequest.prepare, before unsupportedParts: it replaces unsupported image media (user attachments, assistant media, and tool-result images with a local payload) with [Image "name" converted to text by provider/model ...] + description. Images the describer cannot serve keep the existing placeholder, so behavior only improves.
  • Descriptions are generated once per image per describing model (in-process cache keyed by content hash) and reused by every later step, compaction, and title request.
  • Describing model resolution: experimental.image_model in config → automatic. Automatic selection is the new Model.image(providerID?), which prefers the session provider's own vision models, then a known cheap vision family anywhere — it never silently picks a stranger's flagship by release order.
  • Nothing is sent when the model already sees images, when no describer is available, or when a description call fails — all degrade to today's placeholder.

Gated behind the image_fallback feature flag (on in local/dev builds, off in releases; OPENCODE_FLAGS=image_fallback or experimental.image_fallback: true enable it elsewhere). The V1 PR's TUI model picker is intentionally not ported — V2 has no per-prompt model plumbing, so the describer is selected through config; a picker can follow if this lands.

How did you verify your code works?

  • New packages/core/test/session-image-describe.test.ts (9 tests): description substitution for user media and tool-result images, request shape (describe prompt, filename, user-message context), per-image caching, passthrough when the model sees images / no describer exists / the flag is off / the describe call fails, configured experimental.image_model preference, and Model.image catalog selection against the real Model layer.
  • Existing session-model-request* and provider-plugin suites pass (they compose SessionModelRequest.layer with a passthrough SessionImageDescribe).
  • bun run check green; client types and openapi.json regenerated with their scripts.

Visual

<svg xmlns="http://www.w3.org/2000/svg" width="240" height="72" viewBox="0 0 240 72">
  <rect x="8" y="16" width="52" height="40" rx="6" fill="none" stroke="currentColor" stroke-opacity="0.6"/>
  <circle cx="24" cy="30" r="3" fill="currentColor"/>
  <path d="M12 52 L28 38 L38 46 L46 40 L56 50" fill="none" stroke="currentColor" stroke-width="2" stroke-opacity="0.8"/>
  <rect x="8" y="16" width="52" height="40" rx="6" fill="currentColor" opacity="0">
    <animate attributeName="opacity" values="0;0;0.15;0" dur="3s" repeatCount="indefinite" keyTimes="0;0.2;0.6;1"/>
  </rect>
  <path d="M62 36 H82" stroke="currentColor" stroke-width="2" stroke-dasharray="3 3" stroke-opacity="0.7"/>
  <circle cx="96" cy="36" r="10" fill="none" stroke="currentColor" stroke-opacity="0.25"/>
  <path d="M96 26 a10 10 0 0 1 10 10" fill="none" stroke="currentColor" stroke-width="2">
    <animateTransform attributeName="transform" type="rotate" from="0 96 36" to="360 96 36" dur="1.2s" repeatCount="indefinite"/>
  </path>
  <path d="M110 36 H130" stroke="currentColor" stroke-width="2" stroke-dasharray="3 3" stroke-opacity="0.7"/>
  <rect x="140" y="20" width="80" height="6" rx="3" fill="currentColor" opacity="0.15">
    <animate attributeName="opacity" values="0.15;0.85;0.85;0.15" dur="3s" repeatCount="indefinite" keyTimes="0;0.3;0.8;1"/>
  </rect>
  <rect x="140" y="34" width="64" height="6" rx="3" fill="currentColor" opacity="0.15">
    <animate attributeName="opacity" values="0.15;0.15;0.85;0.15" dur="3s" repeatCount="indefinite" keyTimes="0;0.4;0.7;1"/>
  </rect>
  <rect x="140" y="48" width="72" height="6" rx="3" fill="currentColor" opacity="0.15">
    <animate attributeName="opacity" values="0.15;0.15;0.85;0.15" dur="3s" repeatCount="indefinite" keyTimes="0;0.5;0.8;1"/>
  </rect>
</svg>

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@SOUMITRO-SAHA SOUMITRO-SAHA mentioned this pull request Nov 7, 2026
3 of 6 tasks
@opencodejr opencodejr Bot closed this Nov 7, 2026
@opencodejr

opencodejr Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for porting this to V2, @SOUMITRO-SAHA!

I think this duplicates #54366, an open V2 pull request for the same feature. When the session model can't see images, it describes them with a vision-capable model set in experimental config and uses that text instead of the "does not support image input" placeholder. It also caches the descriptions and keeps today's placeholder when a description fails. It has already been through a review round.

If your version has something #54366 lacks, such as picking a vision model automatically from the same provider or the feature flag, please suggest it there or help review and test it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant