feat(core): discover context limits for custom OpenAI-compatible providers by woodsonl · Pull Request #59520 · anomalyco/opencode

woodsonl · 2026-10-27T05:31:27Z

Recreates #54398 (closed when its head fork was deleted; branch restored from the preserved PR head commit 20f9579).

Issue for this PR

Closes #53235

Type of change

New feature

What does this PR do?

Ports #53236 to V2, following the invitation in that PR to reopen against the V2 line.

Custom OpenAI-compatible providers are not in models.dev, so their models keep the built-in placeholder limits and auto-compaction fires against a guess. A new provider plugin requests GET {baseURL}/models once per provider endpoint (4s timeout, 5 minute success cache, 60 second failure retry) and fills context, input, and output from advertised context_length, max_input_tokens, and max_output_tokens.

Adaptations for V2:

The catalog moved from per-model api records to provider package plus merged model overlays. Discovery now applies to models whose effective package is @opencode/ai/providers/openai-compatible, with the base URL and auth resolved from provider and model settings the same way the model snapshot merges them.
V2 model transforms are synchronous, so discovery runs beside the catalog and reloads it once results land, instead of fetching inside the transform.
V2 config-defined models start from the Model.Info default limits rather than zero, so a field is filled when it is absent, zero, or still equal to that default placeholder. Real limits from models.dev and the dedicated LM Studio, Ollama, vLLM, and Modal plugins are never replaced. Configured limit fields win per field through the config provider transform, which is registered after this plugin.
A failed lookup keeps the last successful advertisement and retries after 60 seconds, matching how local provider plugins hold their inventory through outages.
Review scope: only models synthesized without a catalog entry (no release date, no cost rows) are probed or filled. Bundled catalog providers, config models inherited from a canonical catalog provider, and integration models with their own inventory keep their real values, so the placeholder-value comparison is no longer what protects them.
The probe authorizes with the provider's active connection credential, resolved once per discovery cycle, falling back to settings.apiKey; a configured authorization header still wins.

How did you verify your code works?

9 new tests in packages/core/test/plugin/provider-openai-compatible-limits.test.ts: parsing and merge rules, plus plugin-level tests against Bun-served endpoints covering fills, bearer auth, configured limits winning, and fail-open behavior on an erroring endpoint.
bun run test in packages/core: 6601 pass, 2 fail, both reproduced on a clean upstream/dev checkout (session-diff and supervisor-reload flakes, unrelated to this change).
bun run check from the repository root: clean.
bun typecheck in packages/core: clean.

Screenshots / recordings

N/A (no UI change)

Checklist

I have tested my changes locally
I have not included unrelated changes in this PR

Added by the opencode bot (AI-generated).

Fixes #40908

opencode-agent

Works as described. Against a local mock /v1/models, the base keeps the default 200000/32000 limits for a config-defined OpenAI-compatible model and never calls the endpoint. With this PR the model gets the advertised context, input and output, and the configured API key is sent once as a bearer token. The new tests and the packages/core typecheck pass; the same tests also pass with an aisdk:@ai-sdk/openai-compatible package.

Public API: unchanged (adds a built-in provider plugin and a docs section; no new config keys, events or endpoints).

Non-blocking points, inline: models from the LM Studio, Ollama and vLLM plugins are probed as well; the 60s retry and 5-minute refresh only happen at the next catalog change; and the cache key ignores the credential. Also, #57161 fetches the same /models endpoint to add missing models. If both land, sharing one fetch would avoid hitting each endpoint twice per refresh.

Fixes #40908

opencode-agent · 2026-10-27T05:36:25Z

+      endpoints.clear()
+      for (const item of models.provider.list()) {
+        for (const [id, definition] of item.models) {
+          if ((definition.package ?? item.provider.package) !== OPENAI_COMPATIBLE) continue


The LM Studio, Ollama and vLLM plugins (plugin/provider/local.ts) set the provider package to @opencode/ai/providers/openai-compatible and build their models from Model.Info.default, with no release date or cost. Those models pass both fromCatalog checks, so this plugin sends those servers an extra /models request on every cycle. It can also fill their output/input placeholders, even though the description says models from these plugins keep their own values. Could providers that a dedicated discovery plugin owns be skipped here?

Fixed in dafb91b: createLocalProviderPlugin now registers its provider ID in an exported localProviderIDs set, and the limits transform skips those providers entirely — no probe, no fills, so nothing owned by a dedicated discovery plugin gets a second /models request or its placeholders overwritten. Any future provider built on the factory is covered automatically, and since a user-configured provider with one of those IDs is also owned by that plugin (it folds config settings for its own ID), skipping by ID matches ownership exactly. Modal models always carry cost rows, so fromCatalog already kept them out. Added a test configuring vllm through the same openai-compatible package: no request reaches the server and the placeholder limits stay — it fails on the previous head.

opencode-agent · 2026-10-27T05:36:25Z

+    })
+
+    yield* discover().pipe(Effect.ignore, Effect.forkScoped)
+    yield* bus


Discovery only runs at startup and on Model.Event.Updated. The 60s failure retry and 5-minute refresh therefore only happen at the next catalog change, not on a timer. An endpoint that was down at startup stays on placeholder limits until something else reloads the catalog. Either add a schedule (as the local provider plugins do) or adjust the description and docs.

Fixed in dafb91b: discovery now repeats on Schedule.spaced (60s, the failure TTL) like the local provider plugins, so failed endpoints retry about once a minute and successes refresh at their 5-minute TTL on their own — an endpoint down at startup catches up without waiting for a catalog change. The plugin also subscribes to Credential.Event.Switched for its providers so a sign-in revalidates at once. To keep the schedule from churning the catalog, ctx.model.reload() now fires only when the advertisement actually changed. Docs and the module comment state the cadence.

opencode-agent · 2026-10-27T05:36:25Z

+// Shared across Locations so every catalog rebuild reuses one fetch per endpoint.
+const cache = new Map<string, CacheEntry>()
+
+const endpointKey = (endpoint: Endpoint) => `${endpoint.providerID}\n${endpoint.baseURL}`


This module-level cache is keyed by provider ID and base URL only. After the key or connection changes (for example a 401 fixed by opencode auth login), the old result is served until the TTL expires. Including a hash of the authorization in the key, as local.ts does with apiKey, would avoid that.

Fixed on both axes in dafb91b. The cache key now includes a digest of the auth known synchronously (a configured authorization header, else settings.apiKey), like local.ts keys by apiKey. The active connection can't be resolved from the synchronous transform, so each entry instead records the fingerprint of the connection it was fetched under — credential:<id>, plus the key value for key credentials so an in-place edit also invalidates; OAuth access-token rotation deliberately does not, since a refreshed token is not a new account. Discovery compares that fingerprint before trusting an entry, so opencode auth login refetches immediately instead of serving the old account until the TTL expires, and the Credential.Event.Switched subscription makes the switch itself trigger the revalidation. Added a test that rotates settings.apiKey on one endpoint and asserts the refetch and the new bearer — it fails on the previous head.

…iders Custom OpenAI-compatible providers are not in the models.dev catalog, so their models keep the built-in placeholder limits and auto-compaction fires against a guess instead of the real window. Request GET {baseURL}/models once per endpoint and fill context, input, and output from the advertised context_length, max_input_tokens, and max_output_tokens when those fields are still placeholders. Configured limits win per field through the config provider transform that runs after this plugin, and limits from models.dev or dedicated discovery plugins are never replaced. Discovery fails open: an unreachable or malformed endpoint keeps the last known limits and never blocks the catalog. Ports anomalyco#53236 to V2.

…via connections Discovery matched every provider on the openai-compatible package, so bundled catalog providers (zen free models, models.dev entries mapped to the generic package) were probed on every startup. It also authorized only from settings.apiKey or a configured header, but v2 keys live in connections and resolve at request time, so connection-authenticated gateways got a 401 and never filled limits. Eligible models are now those synthesized without a catalog entry: no release date and no cost rows. Catalog-backed models keep their real values. This also retires the placeholder-value comparison as the only guard: 200000/32000 is a real catalog value for some models, so an equality check cannot tell a default from data. Authorization resolves the provider's active connection credential through ctx.integration.connection, behind a configured authorization header and ahead of settings.apiKey.

An empty settings.apiKey produced a bare "Bearer " authorization header on the /models probe. Treat it as absent, as the local provider does, and cover it with a live test.

opencode-agent

This push rebases onto the latest dev and moves the isRecord import to @opencode/util/record. The new limits test file still passes (10/10), and the type check shows no errors in the files this PR changes. The three earlier inline points (local providers being probed, retry and refresh only happening on catalog changes, and the cache key ignoring the credential) are not addressed yet. Public API: unchanged.

…key cache by auth Review follow-up on the OpenAI-compatible limit discovery plugin: - LM Studio, Ollama, and vLLM providers (any provider a local discovery plugin owns) are no longer probed or filled; their own plugin owns the inventory and already hits their endpoints. - Discovery now repeats on a 60s schedule like the local provider plugins, so failed endpoints retry about once a minute and successes refresh at their 5-minute TTL instead of both waiting for the next catalog change. A reload only fires when the advertisement changed. - The fetch cache is keyed by the auth known in settings (authorization header or apiKey digest), and entries record the connection identity they were fetched under; a rotated key or a credential switch (for example after `opencode auth login`) refetches instead of serving the previous account's result. Credential switch events revalidate at once.

woodsonl · 2026-10-27T07:02:46Z

dafb91b addresses the three inline points: local-plugin providers (LM Studio, Ollama, vLLM — any provider built on createLocalProviderPlugin) are skipped by ID, discovery repeats on a 60s Schedule.spaced so the failure retry and 5-minute refresh actually run, and the fetch cache is keyed by the settings auth plus the active connection's identity so rotated keys and credential switches refetch. Two new plugin-level tests cover the skip and the rotation; both fail on the previous head. bun run typecheck and the test/plugin/ suite (429 tests) are green.

On the non-blocking point about sharing the /models fetch with #57161: left as-is for now since that PR hasn't landed; consolidating the two fetches makes sense once both are in, and the cache here should make that straightforward.

opencode-agent

The new commit addresses all three earlier points: providers owned by the LM Studio, Ollama and vLLM plugins are no longer probed; discovery now also runs on a timer; and the cache is keyed by the settings credential and checked against the active connection, with a refetch when a credential is switched. I checked the timer end to end: with the endpoint down when opencode serve started, the model picked up the advertised limits by itself once the endpoint came up, with no catalog change. That retry took about two minutes rather than one, apparently because a failure recorded just after the timer starts makes the next 60s tick skip; not blocking. The limits tests (12/12) and the LM Studio, Ollama and vLLM tests pass. Public API: unchanged.

Flamefire · 2026-11-02T13:52:23Z

Waiting for this feature. Just checked my endpoint and it has input and output tokens but not context length:

    {
      "id": "MiniMaxAI/MiniMax-M3.1-Flash",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai",
      "mode": "chat",
      "max_input_tokens": 1048576,
      "max_output_tokens": 65536
    },

I'd say it makes sense to derive defaults from this. E.g. context_length = max(input, output) and output/input=context (for missing input/output)

opencode-agent Bot reviewed Oct 27, 2026

View reviewed changes

woodsonl added 3 commits October 27, 2026 00:38

fix(core): send no bearer header for an empty apiKey in limit discovery

8243852

An empty settings.apiKey produced a bare "Bearer " authorization header on the /models probe. Treat it as absent, as the local provider does, and cover it with a live test.

woodsonl force-pushed the openai-compatible-limits branch from 20f9579 to 8243852 Compare October 27, 2026 05:38

opencode-agent Bot reviewed Oct 27, 2026

View reviewed changes

opencode-agent Bot mentioned this pull request Oct 27, 2026

feat(core): discover Unsloth Studio models (local runtime — like Ollama, LM Studio, vLLM) #59557

Closed

opencodejr Bot mentioned this pull request Oct 28, 2026

Feature: Dynamic context length detection for OpenAI-compatible providers #40908

Open

joe-ieta mentioned this pull request Nov 14, 2026

[FEATURE]: Feature Request: Add GUI Options to Edit Custom Providers and Manage Model Lists #33704

Open

1 task done

Navigation Menu

Repository navigation

feat(core): discover context limits for custom OpenAI-compatible providers - #59520

feat(core): discover context limits for custom OpenAI-compatible providers#59520
woodsonl wants to merge 4 commits into
anomalyco:devfrom
woodsonl:openai-compatible-limits

woodsonl commented Oct 27, 2026

Uh oh!

opencode-agent Bot left a comment

Uh oh!

opencode-agent Bot Oct 27, 2026

Uh oh!

woodsonl Oct 27, 2026

Uh oh!

opencode-agent Bot Oct 27, 2026

Uh oh!

woodsonl Oct 27, 2026

Uh oh!

opencode-agent Bot Oct 27, 2026

Uh oh!

woodsonl Oct 27, 2026

Uh oh!

opencode-agent Bot left a comment

Uh oh!

woodsonl commented Oct 27, 2026

Uh oh!

opencode-agent Bot left a comment

Uh oh!

Flamefire commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

woodsonl commented Oct 27, 2026

Issue for this PR

Type of change

What does this PR do?

How did you verify your code works?

Screenshots / recordings

Checklist

Uh oh!

opencode-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

opencode-agent Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

woodsonl Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

opencode-agent Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

woodsonl Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

opencode-agent Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

woodsonl Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

opencode-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

woodsonl commented Oct 27, 2026

Uh oh!

opencode-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Flamefire commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants