Skip to content

fix(paintings): fix base64 image handling across providers#13747

Merged
DeJeune merged 6 commits intomainfrom
fix-newapi
Mar 25, 2026
Merged

fix(paintings): fix base64 image handling across providers#13747
DeJeune merged 6 commits intomainfrom
fix-newapi

Conversation

@Pleasurecruise
Copy link
Copy Markdown
Collaborator

@Pleasurecruise Pleasurecruise commented Mar 23, 2026

What this PR does

Before this PR:

  • NewApi image generation hardcoded response_format: 'b64_json', forcing all responses to base64 regardless of model capability
  • All providers that handle base64 responses stored urls: validFiles.map(f => f.name) — just filenames — causing handleRetry to fail with invalid URLs
  • Chat-generated base64 images were stored as full data:image/...;base64,... URIs inside ImageMessageBlock.url, which got included in subsequent message history and caused HTTP 413 errors ([Bug]: HTTP 413 "Payload Too Large" when sending follow-up messages after a Base64 image generation #12602)

After this PR:

  • NewApi no longer forces response_format: 'b64_json'; the API defaults to returning URLs, with base64 as a transparent fallback
  • Base64 responses in NewApi, Aihubmix, and Ovms now correctly store urls: [] (no retry URL exists for inline base64 data)
  • Chat-generated base64 images are persisted to disk immediately; the block stores a file reference and a local file:// URL instead of the raw data URI
  • ImageGenerationMiddleware now prefers block.file over block.url when collecting images from previous assistant messages

https://linux.do/t/topic/1681973/18
Fixes #12602

Why we need it and why it was done in this way

The following tradeoffs were made:

  • Removing response_format: 'b64_json' from NewApi follows the same pattern as Aihubmix's gpt-image-1 handling — no explicit format is set, and the response handler already supports both URL and base64 paths.
  • Storing urls: [] for base64 responses is semantically correct: there is no remote URL to retry from once the base64 payload is gone.
  • Persisting base64 chat images to disk is the minimal fix for [Bug]: HTTP 413 "Payload Too Large" when sending follow-up messages after a Base64 image generation #12602; it reuses the existing saveBase64Image IPC path without changing any message schema or Redux state shape.

The following alternatives were considered:

  • Keeping response_format: 'b64_json' and fixing only the urls field — rejected because NewApi supports URL responses natively, and URL mode is more efficient.

Breaking changes

None.

Special notes for your reviewer

The imageCallbacks.ts change means that after a base64 image is generated in chat, the ImageMessageBlock will have file set and url as a local file:// path. Existing messages already stored with raw base64 URLs are unaffected (they will continue to display correctly via the existing parseDataUrl path in messageConverter.ts).

Checklist

  • PR: The PR description is expressive enough and will help future contributors
  • Code: Write code that humans can understand and Keep it simple
  • Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
  • Upgrade: Impact of this change on upgrade flows was considered and addressed if required
  • Documentation: Not required — no user-facing feature added, only bug fixes
  • Self-review: I have reviewed my own code before requesting review from others

Release note

Fix base64 image handling in paintings (NewApi, Aihubmix, Ovms): images now return as URLs by default, preventing invalid retry URLs. Fix chat image generation storing large base64 data URIs in message history, which caused HTTP 413 errors on follow-up messages.

- Remove hardcoded response_format b64_json from NewApi image generation,
  allowing the API to return URLs by default and adapting to both formats
- Fix incorrect urls field storage for base64 responses (was storing
  filenames instead of empty array) in NewApi, Aihubmix, and Ovms
- Persist base64-generated chat images to disk in imageCallbacks to
  prevent large data URIs from bloating message history (fixes #12602)
- Update ImageGenerationMiddleware to prefer file reference over inline
  base64 URL when collecting images from previous assistant messages

Signed-off-by: Pleasurecruise <[email protected]>
Copilot AI review requested due to automatic review settings March 23, 2026 17:07
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes oversized base64 image payloads and invalid retry URLs across the paintings feature and chat image generation, reducing request sizes (e.g., preventing HTTP 413) and making provider handling more consistent.

Changes:

  • Stop forcing response_format: 'b64_json' for NewApi image generation (prefer URL responses when available).
  • For base64 image responses in paintings providers, persist files but store urls: [] (no retry URL).
  • For chat-generated base64 images, persist to disk and prefer block.file when reusing prior assistant images.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/renderer/src/services/messageStreaming/callbacks/imageCallbacks.ts Persist base64 chat images to disk and store a file + file:// URL.
src/renderer/src/pages/paintings/config/NewApiConfig.ts Remove response_format option from NewApi model config.
src/renderer/src/pages/paintings/NewApiPage.tsx Remove forced response_format: 'b64_json' and fix base64 retry URL storage.
src/renderer/src/pages/paintings/AihubmixPage.tsx Fix base64 retry URL storage (urls: []).
src/renderer/src/pages/paintings/OvmsPage.tsx Fix base64 retry URL storage (urls: []).
src/renderer/src/aiCore/legacy/middleware/feat/ImageGenerationMiddleware.ts Prefer block.file over block.url when collecting prior assistant images.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add a hoisted mockSavedFile and mock FileManager methods (addFile, getFileUrl) as well as window.api.file.saveBase64Image to simulate saved file behavior in the integration test. Update assertions to expect the image block to contain the saved file object and a file:// URL, and adjust mocks to reflect file save/get behavior.
Copy link
Copy Markdown
Contributor

@cherry-ai-bot cherry-ai-bot bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall.

This patch fixes the base64-image handling path in a coherent way across providers:

  • NewApi stops forcing response_format: 'b64_json', allowing URL responses by default
  • providers that only had inline base64 data now correctly persist urls: [] instead of storing invalid retry targets
  • chat-generated base64 images are saved to disk immediately, so large data URIs no longer get embedded into follow-up message history
  • the middleware now prefers block.file when reconstructing assistant-generated images from prior messages

I also checked the added integration-test coverage for the persisted-file path, and it matches the intended behavior.

I did not find a blocking correctness issue in the diff.

Copy link
Copy Markdown
Collaborator

@GeorgeDong32 GeorgeDong32 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

This PR correctly fixes base64 image handling issues across multiple providers.

✅ What's Fixed

  1. NewApi response_format removal - No longer forces b64_json, allowing API to return URLs by default
  2. Correct urls field for base64 responses - Now stores [] instead of incorrect filename array
  3. Chat image persistence - Base64 images are now saved to disk, fixing #12602 (HTTP 413 errors)
  4. ImageGenerationMiddleware update - Prefers block.file reference over inline base64 URL

📋 Test Coverage

  • Updated integration tests with proper mocks for FileManager and saveBase64Image
  • Tests correctly verify file reference and file:// URL output

💡 Minor Suggestion

See inline comment about hoisting the buildImageBlockFields function for better memory efficiency. This is a non-blocking suggestion.

Verdict

Approved ✅ - Ready to merge.

Comment thread src/renderer/src/services/messageStreaming/callbacks/imageCallbacks.ts Outdated
Pleasurecruise and others added 4 commits March 24, 2026 16:49
Import GenerateImageResponse and replace untyped `any` usages in src/renderer/src/services/messageStreaming/callbacks/imageCallbacks.ts. Tighten types for onImageDelta, onImageGenerated (now optional), and buildImageBlockFields to improve type safety when handling generated images (including base64 persistence). No runtime logic changes.
…13 (#12602)

When a chat model outputs markdown base64 images in text (e.g.
`![image](data:image/jpeg;base64,...)`), the entire multi-MB string was
sent back to the API in subsequent messages, causing 413 Payload Too Large.

- Add `stripMarkdownBase64Images` using indexOf-based string scanning
  (avoids regex OOM on large base64 payloads)
- Fix `convertImageBlockToImagePart` to handle `file.ext` with or
  without leading dot
- Replace `isImageEnhancementModel` with `isVisionModel` for image
  merge logic

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: suyao <[email protected]>
@DeJeune DeJeune merged commit 445208d into main Mar 25, 2026
7 checks passed
@DeJeune DeJeune deleted the fix-newapi branch March 25, 2026 06:04
MyPrototypeWhat pushed a commit that referenced this pull request Mar 30, 2026
### What this PR does

Before this PR:
- NewApi image generation hardcoded `response_format: 'b64_json'`,
forcing all responses to base64 regardless of model capability
- All providers that handle base64 responses stored `urls:
validFiles.map(f => f.name)` — just filenames — causing `handleRetry` to
fail with invalid URLs
- Chat-generated base64 images were stored as full
`data:image/...;base64,...` URIs inside `ImageMessageBlock.url`, which
got included in subsequent message history and caused HTTP 413 errors
(#12602)

After this PR:
- NewApi no longer forces `response_format: 'b64_json'`; the API
defaults to returning URLs, with base64 as a transparent fallback
- Base64 responses in NewApi, Aihubmix, and Ovms now correctly store
`urls: []` (no retry URL exists for inline base64 data)
- Chat-generated base64 images are persisted to disk immediately; the
block stores a `file` reference and a local `file://` URL instead of the
raw data URI
- `ImageGenerationMiddleware` now prefers `block.file` over `block.url`
when collecting images from previous assistant messages

https://linux.do/t/topic/1681973/18
Fixes #12602

### Why we need it and why it was done in this way

The following tradeoffs were made:
- Removing `response_format: 'b64_json'` from NewApi follows the same
pattern as Aihubmix's `gpt-image-1` handling — no explicit format is
set, and the response handler already supports both URL and base64
paths.
- Storing `urls: []` for base64 responses is semantically correct: there
is no remote URL to retry from once the base64 payload is gone.
- Persisting base64 chat images to disk is the minimal fix for #12602;
it reuses the existing `saveBase64Image` IPC path without changing any
message schema or Redux state shape.

The following alternatives were considered:
- Keeping `response_format: 'b64_json'` and fixing only the `urls` field
— rejected because NewApi supports URL responses natively, and URL mode
is more efficient.

### Breaking changes

None.

### Special notes for your reviewer

The `imageCallbacks.ts` change means that after a base64 image is
generated in chat, the `ImageMessageBlock` will have `file` set and
`url` as a local `file://` path. Existing messages already stored with
raw base64 URLs are unaffected (they will continue to display correctly
via the existing `parseDataUrl` path in `messageConverter.ts`).

### Checklist

- [x] PR: The PR description is expressive enough and will help future
contributors
- [x] Code: Write code that humans can understand and Keep it simple
- [x] Refactor: You have left the code cleaner than you found it (Boy
Scout Rule)
- [ ] Upgrade: Impact of this change on upgrade flows was considered and
addressed if required
- [x] Documentation: Not required — no user-facing feature added, only
bug fixes
- [x] Self-review: I have reviewed my own code before requesting review
from others

### Release note

```release-note
Fix base64 image handling in paintings (NewApi, Aihubmix, Ovms): images now return as URLs by default, preventing invalid retry URLs. Fix chat image generation storing large base64 data URIs in message history, which caused HTTP 413 errors on follow-up messages.
```

---------

Signed-off-by: Pleasurecruise <[email protected]>
Signed-off-by: suyao <[email protected]>
Co-authored-by: suyao <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: HTTP 413 "Payload Too Large" when sending follow-up messages after a Base64 image generation

4 participants