Repository navigation
docs(providers): document attachment/modalities model capabilities for custom models - #47482
Closed
aref-alapour wants to merge 1 commit into
Closed
aref-alapour wants to merge 1 commit into
aref-alapour wants to merge 1 commit into
Conversation
…r custom models Custom models default to attachment:false. When images are pasted into a session on such a model they are silently replaced by a text placeholder, which is undocumented and easy to mistake for a provider problem. Document the attachment and modalities.input model options in the custom provider guide with a warning about the default. Refs #47480
github-actions
Bot
added
the
needs:compliance
Contributor
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
Contributor
|
This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window. Feel free to open a new pull request that follows our guidelines. |
github-actions
Bot
removed
the
needs:compliance
3 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the per-model capability options
attachmentandmodalities.inputfor custom provider models in the providers guide, with a warning about the silent-placeholder behavior whenattachmentis not set.Closes half of #47480 (the docs half).
Context
Custom models default to
attachment: false. When images are pasted or attached in a session on such a model, the media part is silently replaced by a plain text placeholder ([Attached image/png: file.png]) before the request is sent — the model never receives the image and nothing warns the user. This is especially confusing when the model behind the endpoint actually supports vision: the model typically "explains" that it cannot see images, and users conclude their provider is broken.I hit exactly this against an OpenAI-compatible gateway with Gemini-class (vision) heads, and the only way to discover the flag was inspecting the bundled model loader. Documenting it prevents the trap.
Changes
packages/web/src/content/docs/providers.mdx:attachment: trueandmodalities.input:::warningcallout describing the silent placeholder substitution defaultEnglish page only — translations can follow their usual flow.
Verification
attachment: cfg.attachment ?? false,input.image: cfg.modalities?.input?.includes("image") ?? ...) on 1.18.25 CLI and the desktop app server (1.18.29).attachment: true(+modalities.input) the model transcribes pasted screenshots natively; without it the stored part is the text placeholder.Happy to extend this with the TUI warning proposed in the issue if maintainers want it server-side or client-side — pointers welcome.