Skip to content

feat(core): generate images and decisions through catalog models - #59420

Merged
rekram1-node merged 8 commits into
generate-media-apisfrom
media-generate
Oct 26, 2026
Merged

rekram1-node merged 8 commits into
generate-media-apisfrom
media-generate

Conversation

@rekram1-node

Copy link
Copy Markdown
Collaborator

Stacked on #59190, which defines the endpoints. This wires them up.

Problem

POST /api/experimental/generate/image and /decisions decode their requests and then always fail with 501. Image and decision models are in the catalog, and since #59380 each one points at a providers/<P>/images or /decisions entrypoint with the same model(modelID, settings) contract as chat, but nothing loads or runs them.

Change

POST /generate/image { model, prompt, images?, n?, aspect?, resolution? }
  Model.available(["image"]) → find the ref     not there: 404, another type: 422
  same settings path as chat                    provider + variant settings, stored key, ${VAR}s
  load providers/<P>/images → model(...)        not an image model: 422
  Image.generate                                429 retried, other provider errors: 502
  → { images (base64), tokens?, native? }
 model-resolver
+  loadCatalogPackage    settings + package loading, shared by every model type
   fromCatalogModel      chat: unchanged, now on top of loadCatalogPackage
+  fromCatalogMedia      image / decision: same package call, checks it got the right kind of model
+  resolveImage / resolveDecision(ref)
 generate
+  image, decisions      run the image and decision clients, map errors onto the endpoint's error table
 server                  501s replaced; question names still checked for duplicates first
 plugin                  ctx.generate.image / decisions (was pinned to text)

Smaller pieces:

  • Image and decision models prefer the API whose entrypoint matches their type. 12 OpenRouter and fastrouter Gemini image models list chat-completions first and the images API second, and would otherwise resolve to the chat package.
  • azure/images, google-vertex/images and the three Cloudflare entrypoints build their own URL, so they join BUILDS_OWN_URL and ignore the catalog's templated base_url.
  • System One responses (OpenRouter, TypeSafe, Zen decisions) now fill usage.cost from the host's reported cost, so decision results return cost.
  • cost is only what the provider reports. Image results have none yet, since per-image prices aren't in the catalog shape.
  • Only rate limits are retried (3 times, backing off from 2s): an image or decision call that reached the provider may already be billed.

Out of scope

  • Images through a chat API (50 image models only list chat-completions). They return 422 until there's an adapter for that.
  • Session request hooks (Azure Entra ID tokens, ChatGPT sign-in) don't run here, the same as /generate/text today.

Verification

  • bun run check passes; protocol and client check:generated pass.
  • packages/core test/generate.test.ts: real catalog-shaped models through the real entrypoints with a fake HTTP client and a recorded OpenRouter System One response:
    • image: request URL, Bearer key from provider settings, model id; base64 result and token usage
    • decisions: choice, score and predicate answers in order, provider cost and tokens
    • a Google image variant's thinkingLevel reaches the request body, and a 400 maps to invalid_request
    • a chat model asked for images and an image model asked for decisions give UnsupportedModelError, an unknown ref gives ModelNotFoundError
  • packages/core: test/model-resolver.test.ts, test/plugin, test/models.test.ts pass (532).
  • packages/server test/generate.test.ts: unknown models give 404 on both routes, duplicate question names give 400.
  • packages/ai, packages/plugin, packages/client: bun test passes.

Visual

<svg xmlns="http://www.w3.org/2000/svg" width="240" height="72" viewBox="0 0 240 72" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round">
  <style>
    .p{opacity:.35}
    .dot{animation:go 2.4s ease-in-out infinite}
    .img{opacity:.25;animation:show 2.4s ease-in-out infinite}
    @keyframes go{0%{transform:translateX(0);opacity:0}15%{opacity:1}55%{transform:translateX(118px);opacity:1}70%,100%{transform:translateX(118px);opacity:0}}
    @keyframes show{0%,55%{opacity:.25}70%,90%{opacity:1}100%{opacity:.25}}
    @media (prefers-reduced-motion:reduce){.dot{animation:none;opacity:0}.img{animation:none;opacity:1}}
  </style>
  <rect x="12" y="20" width="56" height="32" rx="8" class="p"/>
  <path d="M24 32h32M24 40h20" class="p"/>
  <path d="M76 36h86" stroke-dasharray="4 6" class="p"/>
  <circle class="dot" cx="80" cy="36" r="4" fill="currentColor" stroke="none"/>
  <g class="img">
    <rect x="172" y="14" width="56" height="44" rx="8"/>
    <circle cx="188" cy="28" r="4"/>
    <path d="M176 52l16-14 12 10 8-6 12 10"/>
  </g>
</svg>

@rekram1-node
rekram1-node merged commit 88bc9e3 into generate-media-apis Oct 26, 2026
18 checks passed
@rekram1-node
rekram1-node deleted the media-generate branch October 26, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants