feat(core): transcribe images for models without vision support by savasturkoglu1 · Pull Request #54366 · anomalyco/opencode

savasturkoglu1 · 2026-10-10T19:31:35Z

Issue for this PR

Type of change

Bug fix
New feature
Refactor / code improvement
Documentation

What does this PR do?

When the selected model cannot read images, V2 replaces each attached image with ERROR: Cannot read "<file>" ... Information the user. This lets the user opt into transcribing those images instead, so they get usable context without switching models.

Set experimental.vision_model to a provider/model that accepts image input. Before each step builds its request, the runner replaces unreadable image attachments with a synthetic transcript message and drops those files from the prompt:

runStep
  SessionContext.load
  SessionVision.bridge            # only when experimental.vision_model is set
    selected model reads images? -> no-op
    transcribe each image (concurrency 2, cached per message + position)
      SessionModelRequest.generate -> llm.stream
    user message minus transcribed files + synthetic "[Image: name]\n<transcript>"
  SessionModelRequest.baseTranscript -> request
    unsupportedParts (nothing left to reject)

It is opt-in: nothing is sent anywhere unless vision_model is configured, and provider usage is recorded on the session with source vision. Failed or empty transcriptions are logged and remembered so they are not retried on every later step; the original error text is still shown in that case. Transcription goes through the normal request path (SessionModelRequest.generate), so it carries the session headers, user agent, and model request hooks.

Touches packages/schema (config field, usage source), packages/core (new SessionVision service, runner hook), and regenerated client types.

How did you verify your code works?

bun test in packages/core/test/session-vision.test.ts covers: transcription and synthetic injection, no-op without config, no-op when the selected model reads images, no-op with no images, caching, failed transcription remembered, and empty transcript treated as failure. Also ran the full packages/core suite and bun run check from the root.

Screenshots / recordings

Not a UI change.

Checklist

I have tested my changes locally
I have not included unrelated changes in this PR

github-actions · 2026-10-10T19:31:58Z

The following comment was made by an LLM, it may be inaccurate:

opencode-agent

The idea is reasonable, but this needs a rebase and some rework before it can land. I ran the new test file (7 pass), the rest of the session-runner* tests (137 pass) and packages/core typecheck (clean). I couldn't run it end to end: that needs a real vision model on the same provider, and the simulated model can't provide one.

Public API: adds the optional experimental.vision_model config key (provider/model), which is backwards compatible. The internal SessionRunnerModel interface gains resolveInfo; nothing outside packages/core changes.

Concerns:

Rebase onto the current runner. On latest dev, requests go through SessionModelRequest (packages/core/src/session/model-request.ts, called from runner/step.ts). It already replaces images the model can't read with an ERROR: Cannot read "<file>" … Inform the user. text, so turns no longer fail with a provider error. The runTurnAttempt / toLLMMessages(context, model) code this hooks into is gone. The bridge needs to run before unsupportedParts/boundImages. The transcription request should also go through that same request-building path, so it gets the session headers, user agent and plugin model.request hooks. Right now it is a separate llm.stream call that skips them.
Images go to another model by default. With no config set, the user's images are sent to an auto-picked model the user never chose, which may cost more. Those tokens also don't show up in the session's cost or token totals. Please make this opt-in through experimental.vision_model, or at least record the usage.
Failures are retried on every step (see inline).
Tests: session-runner-vision.test.ts relies on shared module-level state (let catalogModels, configuredVisionModel, transcribeChunks, a shared requests array) reset by hand in each case. Please build that state inside each test's layer instead, so the cases don't depend on running in order.

Fixes #48332
Related: #46794, #48333, #39824

opencode-agent · 2026-10-10T19:36:15Z

+          Stream.filter(LLMEvent.is.textDelta),
+          Stream.map((event) => event.text),
+          Stream.mkString,
+          Effect.catch(() => Effect.succeed("")),


Errors are swallowed without a log, and an empty or failed result is never cached (line 77). If the vision model is broken or rate-limited, every image gets sent to it again on every step of the turn and on every later turn, each time costing a full request. Please log the failure, and remember failed keys for at least the rest of the turn. Also, this Map lives for the whole process and never drops entries.

opencode-agent · 2026-10-10T19:36:15Z

+      // Only the selected model's own provider is used. Sending the user's image
+      // to a different provider would be surprising, so no candidate means the
+      // images are left untouched and the provider error surfaces unchanged.
+      const candidates = (yield* deps.catalog.model.available()).filter(


With no experimental.vision_model set, this picks the newest vision model on the provider and sends the user's images to it automatically. That model can be much more expensive than the one the user selected, and its usage never reaches the session's cost or token totals. I think this should be opt-in (only when vision_model is configured), or its spend should at least be recorded on the session.

opencode-agent · 2026-10-10T19:36:16Z

      const model = yield* models.resolve(session)
      const entries = yield* SessionHistory.entriesForRunner(db, session.id, system.baselineSeq)
      const context = entries.map((entry) => entry.message)
+      const bridged = yield* vision.bridge({ session, model, messages: context })


On latest dev this code no longer exists. Requests are built by SessionModelRequest (called from runner/step.ts), which already replaces images the model can't read via unsupportedParts. After rebasing, the bridge has to run before that step, or the images will already have been replaced with the error text.

savasturkoglu1 · 2026-10-10T20:33:54Z

Hi, quick context for reviewers:

This ports #48334 to V2, as suggested when that PR was closed ("no longer taking PRs for V1, please port it over").

Before: in V2, prompt attachments are lowered into provider media parts unconditionally, so pasting a screenshot while a non-vision model is selected sends the image to the provider and fails the turn with a provider error.

After: the image is transcribed by an active vision-capable model from the same provider (or experimental.vision_model if set) and injected as a synthetic text message, so the selected model and the visible timeline stay unchanged. Any failure falls back to today's behavior. Cross-provider fallback is intentionally still out of scope; local OCR remains the future path there.

This is scoped as a first slice of the runner's unchecked "attachment normalization" item — it doesn't attempt full materialization, and to keep the review small it reuses SessionRunnerModel for credential-backed model resolution rather than duplicating it.

Happy to adjust the approach — e.g. preferring the catalog's small model for transcription, or a different config shape. (The typecheck/test workflows are likely waiting for approval again for this first-time contributor account.)

savasturkoglu1 · 2026-10-12T06:21:32Z

Thanks for the review — rebased onto latest dev and reworked along all four points.

Runner seam. The bridge now runs in runner/llm.ts runStep, on the loaded SessionMessage.Info[] right before SessionModelRequest.baseTranscript, so unsupportedParts sees no unreadable image media left to replace. Transcription goes through SessionModelRequest.generate(...), so it carries the same session headers, user agent, and session/model.request hooks as other requests.

Opt-in. Nothing is transcribed unless experimental.vision_model is set. The auto-pick is gone. Usage is also recorded on the session as source: "vision" (added to session.usage.recorded, client regenerated).

Failures. A failed or empty transcription is logged (Effect.logWarning) and remembered as a failed key, so it is not retried on every later step or turn. The cache is bounded (256 entries, FIFO eviction) instead of living forever.

Tests. packages/core/test/session-vision.test.ts no longer uses shared module-level state; each case builds its own service via SessionVision.layerWith(...) and holds its counters in the test closure.

PR description. Updated to the repository template.

bun test in packages/core and bun run check from the root are green. Happy to adjust the config shape or drop usage recording if you would rather keep that as a follow-up.

github-actions · 2026-10-12T06:21:36Z

Thanks for updating your PR! It now meets our contributing guidelines. 👍

github-actions · 2026-11-10T22:17:37Z

Automated PR Cleanup

Thank you for contributing to opencode.

Due to the high volume of PRs from users and AI agents, we periodically close older PRs using automated criteria so maintainers can focus review time on the most active and community-supported contributions.

This PR was closed because it matched the following cleanup criteria:

The PR was created more than 1 month ago
The PR had fewer than 2 positive reactions
Positive reactions are counted as thumbs-up, heart, celebration, or rocket reactions on the PR

PRs created within the last month are not affected by this cleanup.

If you believe this PR was closed incorrectly, or if you are still actively working on it, please leave a comment explaining why it should be reopened. A maintainer can review and reopen it if appropriate.

Thanks again for taking the time to contribute.

github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Oct 10, 2026

opencode-agent Bot reviewed Oct 10, 2026

View reviewed changes

savasturkoglu1 force-pushed the vision-transcribe-v2 branch from 9b3a09d to 4ffaf9f Compare October 12, 2026 06:21

github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Oct 12, 2026

feat(core): transcribe images for models without vision support

6b233dd

savasturkoglu1 force-pushed the vision-transcribe-v2 branch from 4ffaf9f to 6b233dd Compare October 12, 2026 09:18

This was referenced Oct 12, 2026

ERROR: Cannot read "X" (this model does not support image input). Inform the user. #54887

Closed

sessions with image parts in history wedge forever in an endless 'model does not support images' loop #55309

Closed

savasturkoglu1 added 5 commits October 14, 2026 16:59

Merge branch 'dev' into vision-transcribe-v2

698f8bc

Merge branch 'dev' into vision-transcribe-v2

02f8ba9

Merge branch 'dev' into vision-transcribe-v2

cc5e7c2

Merge branch 'dev' into vision-transcribe-v2

848ec80

Merge branch 'dev' into vision-transcribe-v2

fd592b3

opencode-agent Bot mentioned this pull request Oct 23, 2026

opencode paste image lost prompt #58291

Closed

This was referenced Nov 7, 2026

provider: image attachment to text-only model poisons whole session (error repeats every turn) #62503

Closed

feat(core): describe images with a vision model when the session model cannot see them #62534

Closed

github-actions Bot closed this Nov 10, 2026

github-actions Bot added the automated-pr-cleanup label Nov 10, 2026

Navigation Menu

Repository navigation

feat(core): transcribe images for models without vision support - #54366

feat(core): transcribe images for models without vision support#54366
savasturkoglu1 wants to merge 6 commits into
anomalyco:devfrom
savasturkoglu1:vision-transcribe-v2

savasturkoglu1 commented Oct 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

opencode-agent Bot left a comment

Uh oh!

opencode-agent Bot Oct 10, 2026

Uh oh!

opencode-agent Bot Oct 10, 2026

Uh oh!

opencode-agent Bot Oct 10, 2026

Uh oh!

savasturkoglu1 commented Oct 10, 2026

Uh oh!

savasturkoglu1 commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

savasturkoglu1 commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue for this PR

Type of change

What does this PR do?

How did you verify your code works?

Screenshots / recordings

Checklist

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

opencode-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

opencode-agent Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

opencode-agent Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

opencode-agent Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

savasturkoglu1 commented Oct 10, 2026

Uh oh!

savasturkoglu1 commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

savasturkoglu1 commented Oct 10, 2026 •

edited

Loading