Repository navigation
feat(core): describe images with a vision model when the session model cannot see them - #62534
Closed
SOUMITRO-SAHA wants to merge 1 commit into
Closed
SOUMITRO-SAHA wants to merge 1 commit into
SOUMITRO-SAHA wants to merge 1 commit into
Conversation
…l cannot see them
3 of 6 tasks
Contributor
|
Thanks for porting this to V2, @SOUMITRO-SAHA! I think this duplicates #54366, an open V2 pull request for the same feature. When the session model can't see images, it describes them with a vision-capable model set in experimental config and uses that text instead of the "does not support image input" placeholder. It also caches the descriptions and keeps today's placeholder when a description fails. It has already been through a review round. If your version has something #54366 lacks, such as picking a vision model automatically from the same provider or the feature flag, please suggest it there or help review and test it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Ports #26164 to V2. That PR (and its issue #26160) were closed as V1-only with the ask to port the feature to V2.
Type of change
What does this PR do?
When the session model cannot see images, V2 currently replaces every image with
ERROR: Cannot read "x.png" (this model does not support image input). Inform the user.(viaunsupportedPartsinSessionModelRequest). This PR instead describes the image once with a vision-capable model and substitutes that description, so the conversation can continue on a text-only model.SessionImageDescribeservice runs insideSessionModelRequest.prepare, beforeunsupportedParts: it replaces unsupported image media (user attachments, assistant media, and tool-result images with a local payload) with[Image "name" converted to text by provider/model ...]+ description. Images the describer cannot serve keep the existing placeholder, so behavior only improves.experimental.image_modelin config → automatic. Automatic selection is the newModel.image(providerID?), which prefers the session provider's own vision models, then a known cheap vision family anywhere — it never silently picks a stranger's flagship by release order.Gated behind the
image_fallbackfeature flag (on in local/dev builds, off in releases;OPENCODE_FLAGS=image_fallbackorexperimental.image_fallback: trueenable it elsewhere). The V1 PR's TUI model picker is intentionally not ported — V2 has no per-prompt model plumbing, so the describer is selected through config; a picker can follow if this lands.How did you verify your code works?
packages/core/test/session-image-describe.test.ts(9 tests): description substitution for user media and tool-result images, request shape (describe prompt, filename, user-message context), per-image caching, passthrough when the model sees images / no describer exists / the flag is off / the describe call fails, configuredexperimental.image_modelpreference, andModel.imagecatalog selection against the real Model layer.session-model-request*and provider-plugin suites pass (they composeSessionModelRequest.layerwith a passthroughSessionImageDescribe).bun run checkgreen; client types andopenapi.jsonregenerated with their scripts.Visual
Checklist