fix(paintings): fix base64 image handling across providers#13747
fix(paintings): fix base64 image handling across providers#13747
Conversation
- Remove hardcoded response_format b64_json from NewApi image generation, allowing the API to return URLs by default and adapting to both formats - Fix incorrect urls field storage for base64 responses (was storing filenames instead of empty array) in NewApi, Aihubmix, and Ovms - Persist base64-generated chat images to disk in imageCallbacks to prevent large data URIs from bloating message history (fixes #12602) - Update ImageGenerationMiddleware to prefer file reference over inline base64 URL when collecting images from previous assistant messages Signed-off-by: Pleasurecruise <[email protected]>
There was a problem hiding this comment.
Pull request overview
Fixes oversized base64 image payloads and invalid retry URLs across the paintings feature and chat image generation, reducing request sizes (e.g., preventing HTTP 413) and making provider handling more consistent.
Changes:
- Stop forcing
response_format: 'b64_json'for NewApi image generation (prefer URL responses when available). - For base64 image responses in paintings providers, persist files but store
urls: [](no retry URL). - For chat-generated base64 images, persist to disk and prefer
block.filewhen reusing prior assistant images.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/renderer/src/services/messageStreaming/callbacks/imageCallbacks.ts | Persist base64 chat images to disk and store a file + file:// URL. |
| src/renderer/src/pages/paintings/config/NewApiConfig.ts | Remove response_format option from NewApi model config. |
| src/renderer/src/pages/paintings/NewApiPage.tsx | Remove forced response_format: 'b64_json' and fix base64 retry URL storage. |
| src/renderer/src/pages/paintings/AihubmixPage.tsx | Fix base64 retry URL storage (urls: []). |
| src/renderer/src/pages/paintings/OvmsPage.tsx | Fix base64 retry URL storage (urls: []). |
| src/renderer/src/aiCore/legacy/middleware/feat/ImageGenerationMiddleware.ts | Prefer block.file over block.url when collecting prior assistant images. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add a hoisted mockSavedFile and mock FileManager methods (addFile, getFileUrl) as well as window.api.file.saveBase64Image to simulate saved file behavior in the integration test. Update assertions to expect the image block to contain the saved file object and a file:// URL, and adjust mocks to reflect file save/get behavior.
There was a problem hiding this comment.
LGTM overall.
This patch fixes the base64-image handling path in a coherent way across providers:
- NewApi stops forcing
response_format: 'b64_json', allowing URL responses by default - providers that only had inline base64 data now correctly persist
urls: []instead of storing invalid retry targets - chat-generated base64 images are saved to disk immediately, so large data URIs no longer get embedded into follow-up message history
- the middleware now prefers
block.filewhen reconstructing assistant-generated images from prior messages
I also checked the added integration-test coverage for the persisted-file path, and it matches the intended behavior.
I did not find a blocking correctness issue in the diff.
GeorgeDong32
left a comment
There was a problem hiding this comment.
Code Review Summary
This PR correctly fixes base64 image handling issues across multiple providers.
✅ What's Fixed
- NewApi response_format removal - No longer forces b64_json, allowing API to return URLs by default
- Correct urls field for base64 responses - Now stores [] instead of incorrect filename array
- Chat image persistence - Base64 images are now saved to disk, fixing #12602 (HTTP 413 errors)
- ImageGenerationMiddleware update - Prefers block.file reference over inline base64 URL
📋 Test Coverage
- Updated integration tests with proper mocks for FileManager and saveBase64Image
- Tests correctly verify file reference and file:// URL output
💡 Minor Suggestion
See inline comment about hoisting the buildImageBlockFields function for better memory efficiency. This is a non-blocking suggestion.
Verdict
Approved ✅ - Ready to merge.
Import GenerateImageResponse and replace untyped `any` usages in src/renderer/src/services/messageStreaming/callbacks/imageCallbacks.ts. Tighten types for onImageDelta, onImageGenerated (now optional), and buildImageBlockFields to improve type safety when handling generated images (including base64 persistence). No runtime logic changes.
…13 (#12602) When a chat model outputs markdown base64 images in text (e.g. ``), the entire multi-MB string was sent back to the API in subsequent messages, causing 413 Payload Too Large. - Add `stripMarkdownBase64Images` using indexOf-based string scanning (avoids regex OOM on large base64 payloads) - Fix `convertImageBlockToImagePart` to handle `file.ext` with or without leading dot - Replace `isImageEnhancementModel` with `isVisionModel` for image merge logic Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: suyao <[email protected]>
### What this PR does Before this PR: - NewApi image generation hardcoded `response_format: 'b64_json'`, forcing all responses to base64 regardless of model capability - All providers that handle base64 responses stored `urls: validFiles.map(f => f.name)` — just filenames — causing `handleRetry` to fail with invalid URLs - Chat-generated base64 images were stored as full `data:image/...;base64,...` URIs inside `ImageMessageBlock.url`, which got included in subsequent message history and caused HTTP 413 errors (#12602) After this PR: - NewApi no longer forces `response_format: 'b64_json'`; the API defaults to returning URLs, with base64 as a transparent fallback - Base64 responses in NewApi, Aihubmix, and Ovms now correctly store `urls: []` (no retry URL exists for inline base64 data) - Chat-generated base64 images are persisted to disk immediately; the block stores a `file` reference and a local `file://` URL instead of the raw data URI - `ImageGenerationMiddleware` now prefers `block.file` over `block.url` when collecting images from previous assistant messages https://linux.do/t/topic/1681973/18 Fixes #12602 ### Why we need it and why it was done in this way The following tradeoffs were made: - Removing `response_format: 'b64_json'` from NewApi follows the same pattern as Aihubmix's `gpt-image-1` handling — no explicit format is set, and the response handler already supports both URL and base64 paths. - Storing `urls: []` for base64 responses is semantically correct: there is no remote URL to retry from once the base64 payload is gone. - Persisting base64 chat images to disk is the minimal fix for #12602; it reuses the existing `saveBase64Image` IPC path without changing any message schema or Redux state shape. The following alternatives were considered: - Keeping `response_format: 'b64_json'` and fixing only the `urls` field — rejected because NewApi supports URL responses natively, and URL mode is more efficient. ### Breaking changes None. ### Special notes for your reviewer The `imageCallbacks.ts` change means that after a base64 image is generated in chat, the `ImageMessageBlock` will have `file` set and `url` as a local `file://` path. Existing messages already stored with raw base64 URLs are unaffected (they will continue to display correctly via the existing `parseDataUrl` path in `messageConverter.ts`). ### Checklist - [x] PR: The PR description is expressive enough and will help future contributors - [x] Code: Write code that humans can understand and Keep it simple - [x] Refactor: You have left the code cleaner than you found it (Boy Scout Rule) - [ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required - [x] Documentation: Not required — no user-facing feature added, only bug fixes - [x] Self-review: I have reviewed my own code before requesting review from others ### Release note ```release-note Fix base64 image handling in paintings (NewApi, Aihubmix, Ovms): images now return as URLs by default, preventing invalid retry URLs. Fix chat image generation storing large base64 data URIs in message history, which caused HTTP 413 errors on follow-up messages. ``` --------- Signed-off-by: Pleasurecruise <[email protected]> Signed-off-by: suyao <[email protected]> Co-authored-by: suyao <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>
What this PR does
Before this PR:
response_format: 'b64_json', forcing all responses to base64 regardless of model capabilityurls: validFiles.map(f => f.name)— just filenames — causinghandleRetryto fail with invalid URLsdata:image/...;base64,...URIs insideImageMessageBlock.url, which got included in subsequent message history and caused HTTP 413 errors ([Bug]: HTTP 413 "Payload Too Large" when sending follow-up messages after a Base64 image generation #12602)After this PR:
response_format: 'b64_json'; the API defaults to returning URLs, with base64 as a transparent fallbackurls: [](no retry URL exists for inline base64 data)filereference and a localfile://URL instead of the raw data URIImageGenerationMiddlewarenow prefersblock.fileoverblock.urlwhen collecting images from previous assistant messageshttps://linux.do/t/topic/1681973/18
Fixes #12602
Why we need it and why it was done in this way
The following tradeoffs were made:
response_format: 'b64_json'from NewApi follows the same pattern as Aihubmix'sgpt-image-1handling — no explicit format is set, and the response handler already supports both URL and base64 paths.urls: []for base64 responses is semantically correct: there is no remote URL to retry from once the base64 payload is gone.saveBase64ImageIPC path without changing any message schema or Redux state shape.The following alternatives were considered:
response_format: 'b64_json'and fixing only theurlsfield — rejected because NewApi supports URL responses natively, and URL mode is more efficient.Breaking changes
None.
Special notes for your reviewer
The
imageCallbacks.tschange means that after a base64 image is generated in chat, theImageMessageBlockwill havefileset andurlas a localfile://path. Existing messages already stored with raw base64 URLs are unaffected (they will continue to display correctly via the existingparseDataUrlpath inmessageConverter.ts).Checklist
Release note