Skip to content

Commit dafb91b

Browse files
committed
fix(core): skip dedicated local providers, schedule limit discovery, key cache by auth
Review follow-up on the OpenAI-compatible limit discovery plugin: - LM Studio, Ollama, and vLLM providers (any provider a local discovery plugin owns) are no longer probed or filled; their own plugin owns the inventory and already hits their endpoints. - Discovery now repeats on a 60s schedule like the local provider plugins, so failed endpoints retry about once a minute and successes refresh at their 5-minute TTL instead of both waiting for the next catalog change. A reload only fires when the advertisement changed. - The fetch cache is keyed by the auth known in settings (authorization header or apiKey digest), and entries record the connection identity they were fetched under; a rotated key or a credential switch (for example after `opencode auth login`) refetches instead of serving the previous account's result. Credential switch events revalidate at once.
1 parent 8243852 commit dafb91b

4 files changed

Lines changed: 208 additions & 40 deletions

File tree

‎packages/core/src/plugin/provider/local.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,14 @@ export type LocalModel = {
2020
readonly context?: number | null
2121
}
2222

23+
/**
24+
* Provider IDs whose inventory a local discovery plugin owns. Populated by `createLocalProviderPlugin` at module
25+
* load, so generic discovery (for example OpenAI-compatible limit probing) can leave these providers to their
26+
* dedicated plugin instead of hitting their servers a second time.
27+
*/
28+
const ownedProviderIDs = new Set<string>()
29+
export const localProviderIDs: ReadonlySet<string> = ownedProviderIDs
30+
2331
export type LocalDiscoveryClient = ReturnType<typeof createClient>
2432

2533
export function createLocalProviderPlugin<Discovered>(input: {
@@ -34,6 +42,7 @@ export function createLocalProviderPlugin<Discovered>(input: {
3442
const discovery = new Map<string, { checked: number; apiKey?: string; models?: readonly Discovered[] }>()
3543
const discoveryLock = Semaphore.makeUnsafe(1)
3644
const suffix = input.stripPathSuffix ?? /\/v1$/
45+
ownedProviderIDs.add(input.providerID)
3746

3847
return function make(origin?: string, interval: Duration.Input = "30 seconds") {
3948
return define({

‎packages/core/src/plugin/provider/openai-compatible-limits.ts‎

Lines changed: 109 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,26 @@
1+
import { createHash } from "node:crypto"
12
import { isRecord } from "@opencode/util/record"
23
import { define } from "@opencode/plugin/effect/plugin"
3-
import { Effect, Option, Semaphore, Stream } from "effect"
4+
import { Effect, Option, Schedule, Semaphore, Stream } from "effect"
45
import { HttpClient, HttpClientRequest } from "effect/http"
56
import type { Value as CredentialValue } from "@opencode/schema/credential"
67
import { Bus } from "../../bus.js"
8+
import { Credential } from "../../credential.js"
9+
import { IntegrationConnection } from "../../integration/connection.js"
710
import { Model } from "../../model.js"
811
import { Provider } from "../../provider.js"
12+
import { localProviderIDs } from "./local.js"
913

1014
// Fill missing context limits for generic OpenAI-compatible providers from the
1115
// provider's own `GET {baseURL}/models` advertisement.
1216
//
1317
// Custom models synthesized without a models.dev catalog entry carry the
1418
// built-in placeholder limits, so auto-compaction fires against a guess
1519
// instead of the real window. Catalog-backed models — bundled providers,
16-
// config models inherited from a canonical catalog provider, and integration
17-
// models with their own inventory — keep their real values and are neither
18-
// probed nor filled.
20+
// config models inherited from a canonical catalog provider, integration
21+
// models with their own cost rows, and providers a dedicated discovery
22+
// plugin owns (LM Studio, Ollama, vLLM) — keep their real values and are
23+
// neither probed nor filled.
1924
//
2025
// Auth for the probe, highest first: a configured `authorization` header, the
2126
// provider's active connection credential (keys added via `opencode auth` or
@@ -30,6 +35,10 @@ import { Provider } from "../../provider.js"
3035
//
3136
// Fail-open: an unreachable, erroring, or malformed endpoint is skipped and
3237
// keeps its previously discovered limits; discovery never blocks the catalog.
38+
// Discovery runs at startup, whenever the catalog or the active credential
39+
// changes, and on the schedule below, so failed endpoints retry once a
40+
// minute and successes refresh at their TTL instead of waiting for a catalog
41+
// change.
3342

3443
const FETCH_TIMEOUT = "4 seconds"
3544
const CACHE_TTL_MS = 5 * 60_000
@@ -55,13 +64,24 @@ type Endpoint = {
5564
type CacheEntry = {
5665
readonly at: number
5766
readonly ttl: number
67+
/** Fingerprint of the connection credential the cached fetch ran under; `""` when none was active. */
68+
readonly connection: string
5869
readonly value: Record<string, AdvertisedLimit>
5970
}
6071

6172
// Shared across Locations so every catalog rebuild reuses one fetch per endpoint.
6273
const cache = new Map<string, CacheEntry>()
6374

64-
const endpointKey = (endpoint: Endpoint) => `${endpoint.providerID}\n${endpoint.baseURL}`
75+
const digest = (value: string) => createHash("sha256").update(value).digest("hex").slice(0, 16)
76+
77+
// The key covers everything known synchronously: provider, URL, and the auth
78+
// set directly in settings, so a rotated `apiKey` or `authorization` header
79+
// never reuses the previous credential's result. The active connection's
80+
// identity rides in the entry and is compared at discovery time.
81+
const endpointKey = (endpoint: Endpoint) =>
82+
`${endpoint.providerID}\n${endpoint.baseURL}\n${
83+
endpoint.authorization ?? (endpoint.apiKey ? digest(endpoint.apiKey) : "")
84+
}`
6585

6686
const freshEntry = (endpoint: Endpoint) => {
6787
const entry = cache.get(endpointKey(endpoint))
@@ -150,6 +170,17 @@ const bearerFrom = (credential: CredentialValue | undefined) => {
150170
// An empty key is absent, matching local providers, so the probe sends no bare `Bearer ` header.
151171
const bearerKey = (apiKey: string | undefined) => (apiKey ? `Bearer ${apiKey}` : undefined)
152172

173+
// Identifies which connection the cached fetch ran under. The connection's
174+
// identity (not a rotating OAuth access token) keys the cache, plus the stored
175+
// key value for key credentials, which can be edited in place under the same id.
176+
const connectionFingerprint = (
177+
connection: Parameters<typeof IntegrationConnection.key>[0],
178+
credential: CredentialValue | undefined,
179+
) => {
180+
const identity = IntegrationConnection.key(connection) ?? ""
181+
return credential?.type === "key" ? `${identity}:${digest(credential.key)}` : identity
182+
}
183+
153184
export const OpenAICompatibleLimitsPlugin = define({
154185
id: "opencode.provider.openai-compatible-limits",
155186
effect: Effect.fn(function* (ctx) {
@@ -161,6 +192,9 @@ export const OpenAICompatibleLimitsPlugin = define({
161192
yield* ctx.model.transform((models) => {
162193
endpoints.clear()
163194
for (const item of models.provider.list()) {
195+
// A dedicated discovery plugin owns this provider's inventory and probes
196+
// its own endpoints; a second /models request here would be redundant.
197+
if (localProviderIDs.has(item.provider.id)) continue
164198
for (const [id, definition] of item.models) {
165199
if ((definition.package ?? item.provider.package) !== OPENAI_COMPATIBLE) continue
166200
if (fromCatalog(definition)) continue
@@ -201,60 +235,100 @@ export const OpenAICompatibleLimitsPlugin = define({
201235
const discover = Effect.fn("OpenAICompatibleLimits.discover")(function* () {
202236
yield* lock.withPermit(
203237
Effect.gen(function* () {
204-
const stale = [...endpoints.values()].filter((endpoint) => freshEntry(endpoint) === undefined)
238+
// Resolve the active connection first: an entry cached under a
239+
// different credential (for example after `opencode auth login`) is
240+
// stale no matter its TTL.
241+
const stale: Array<{
242+
endpoint: Endpoint
243+
connection: string
244+
authorization: string | undefined
245+
}> = []
246+
for (const endpoint of endpoints.values()) {
247+
const connection = yield* ctx.integration.connection
248+
.active(endpoint.providerID)
249+
.pipe(Effect.orElseSucceed(() => undefined))
250+
const credential = connection
251+
? yield* ctx.integration.connection
252+
.resolve(connection)
253+
.pipe(Effect.orElseSucceed(() => undefined))
254+
: undefined
255+
const fingerprint = connectionFingerprint(connection, credential)
256+
const entry = cache.get(endpointKey(endpoint))
257+
if (entry !== undefined && Date.now() - entry.at < entry.ttl && entry.connection === fingerprint)
258+
continue
259+
stale.push({
260+
endpoint,
261+
connection: fingerprint,
262+
authorization:
263+
endpoint.authorization ?? bearerFrom(credential) ?? bearerKey(endpoint.apiKey),
264+
})
265+
}
266+
if (stale.length === 0) return
205267
const results = yield* Effect.all(
206-
stale.map((endpoint) =>
207-
Effect.option(
208-
ctx.integration.connection
209-
.active(endpoint.providerID)
210-
.pipe(
211-
Effect.flatMap((connection) =>
212-
connection
213-
? ctx.integration.connection
214-
.resolve(connection)
215-
.pipe(Effect.orElseSucceed(() => undefined), Effect.map(bearerFrom))
216-
: Effect.succeed(undefined),
217-
),
218-
Effect.orElseSucceed(() => undefined),
219-
Effect.flatMap((resolved) =>
220-
fetchLimits(
221-
http,
222-
endpoint.baseURL,
223-
endpoint.authorization ?? resolved ?? bearerKey(endpoint.apiKey),
224-
),
225-
),
226-
),
227-
),
228-
),
268+
stale.map((item) => Effect.option(fetchLimits(http, item.endpoint.baseURL, item.authorization))),
229269
{ concurrency: "unbounded" },
230270
)
231271
let changed = false
232-
for (const [index, endpoint] of stale.entries()) {
272+
for (const [index, item] of stale.entries()) {
233273
const result = results[index]
234-
const previous = cache.get(endpointKey(endpoint))
274+
const key = endpointKey(item.endpoint)
275+
const previous = cache.get(key)
235276
if (Option.isNone(result)) {
236277
// Keep the last successful inventory through transient outages and retry soon.
237-
cache.set(endpointKey(endpoint), {
278+
cache.set(key, {
238279
at: Date.now(),
239280
ttl: FAILURE_TTL_MS,
281+
connection: item.connection,
240282
value: previous?.value ?? {},
241283
})
242284
continue
243285
}
244-
cache.set(endpointKey(endpoint), { at: Date.now(), ttl: CACHE_TTL_MS, value: result.value })
245-
changed = true
286+
cache.set(key, {
287+
at: Date.now(),
288+
ttl: CACHE_TTL_MS,
289+
connection: item.connection,
290+
value: result.value,
291+
})
292+
if (previous === undefined || !sameLimits(previous.value, result.value)) changed = true
246293
}
247294
if (changed) yield* ctx.model.reload()
248295
}),
249296
)
250297
})
251298

252-
yield* discover().pipe(Effect.ignore, Effect.forkScoped)
299+
// Failed endpoints retry once a minute and successes refresh at their TTL,
300+
// on a schedule like the local provider plugins, instead of only when the
301+
// catalog changes.
302+
yield* discover().pipe(Effect.ignore, Effect.repeat(Schedule.spaced(FAILURE_TTL_MS)), Effect.forkScoped)
253303
yield* bus
254304
.subscribe([Model.Event.Updated])
255305
.pipe(
256306
Stream.runForEach(() => discover().pipe(Effect.ignore)),
257307
Effect.forkScoped({ startImmediately: true }),
258308
)
309+
// A credential switch can change what /models advertises; revalidate at
310+
// once instead of waiting out the cache TTL.
311+
yield* bus
312+
.subscribe(Credential.Event.Switched)
313+
.pipe(
314+
Stream.filter((event) =>
315+
[...endpoints.values()].some(
316+
(endpoint) => endpoint.providerID === Provider.ID.make(event.data.integrationID),
317+
),
318+
),
319+
Stream.runForEach(() => discover().pipe(Effect.ignore)),
320+
Effect.forkScoped({ startImmediately: true }),
321+
)
259322
}),
260323
})
324+
325+
const sameLimits = (left: Record<string, AdvertisedLimit>, right: Record<string, AdvertisedLimit>) => {
326+
const ids = Object.keys(left)
327+
if (ids.length !== Object.keys(right).length) return false
328+
return ids.every((id) => {
329+
const a = left[id]
330+
const b = right[id]
331+
if (b === undefined) return false
332+
return a.context === b.context && a.input === b.input && a.output === b.output
333+
})
334+
}

‎packages/core/test/plugin/provider-openai-compatible-limits.test.ts‎

Lines changed: 84 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ import {
1010
} from "@opencode/core/plugin/provider/openai-compatible-limits"
1111
import { ProviderPlugins } from "@opencode/core/plugin/provider"
1212
import { Provider } from "@opencode/core/provider"
13-
import { Document, type Entry, Info } from "@opencode/schema/config"
13+
import { Bus } from "@opencode/core/bus"
14+
import { Document, type Entry, Event, Info } from "@opencode/schema/config"
1415
import { describe, expect, test } from "bun:test"
1516
import { Effect, Schema } from "effect"
1617
import { testEffect } from "../lib/effect"
@@ -23,8 +24,11 @@ const addPlugins = Effect.fn(function* (entries: Entry[]) {
2324
const plugin = yield* Plugin.Service
2425
const host = yield* PluginHost.make(plugin)
2526
const config = Config.testLayer(entries)
26-
yield* OpenAICompatibleLimitsPlugin.effect(host).pipe(Effect.provide(config))
27-
yield* ConfigProviderPlugin.Plugin.effect(host).pipe(Effect.provide(config))
27+
return yield* Effect.gen(function* () {
28+
yield* OpenAICompatibleLimitsPlugin.effect(host)
29+
yield* ConfigProviderPlugin.Plugin.effect(host)
30+
return yield* Config.Test
31+
}).pipe(Effect.provide(config))
2832
})
2933

3034
function eventually<A>(
@@ -242,4 +246,81 @@ describe("OpenAICompatibleLimitsPlugin", () => {
242246
({ dead, live }) => Effect.promise(() => Promise.all([dead.stop(true), live.stop(true)])),
243247
),
244248
)
249+
250+
it.live("does not probe providers owned by a dedicated discovery plugin", () =>
251+
Effect.acquireUseRelease(
252+
Effect.sync(() => {
253+
const requests: string[] = []
254+
const server = Bun.serve({
255+
port: 0,
256+
fetch: (request) => {
257+
requests.push(new URL(request.url).pathname)
258+
return Response.json({ data: [{ id: "local-model", context_length: 1_000_000 }] })
259+
},
260+
})
261+
return { requests, server }
262+
}),
263+
({ requests, server }) =>
264+
Effect.gen(function* () {
265+
const modelState = yield* Model.Service
266+
yield* addPlugins([configuration("vllm", `${server.url.origin}/v1`, { "local-model": {} })])
267+
268+
const model = yield* eventually(
269+
modelState.get(Provider.ID.make("vllm"), Model.ID.make("local-model")),
270+
(model) => model !== undefined,
271+
)
272+
// LM Studio, Ollama, and vLLM keep the inventory their own plugin discovers.
273+
yield* Effect.promise(() => Bun.sleep(200))
274+
expect(requests).toEqual([])
275+
expect(model?.limit).toMatchObject({ context: 200_000, output: 32_000 })
276+
}),
277+
({ server }) => Effect.promise(() => server.stop(true)),
278+
),
279+
)
280+
281+
it.live("refetches when the settings apiKey changes instead of serving the cached result", () =>
282+
Effect.acquireUseRelease(
283+
Effect.sync(() => {
284+
const authorizations: Array<string | null> = []
285+
const server = Bun.serve({
286+
port: 0,
287+
fetch: (request) => {
288+
const authorization = request.headers.get("authorization")
289+
authorizations.push(authorization)
290+
return Response.json({
291+
data: [
292+
{
293+
id: "rotating-model",
294+
context_length: authorization === "Bearer one" ? 111_111 : 222_222,
295+
},
296+
],
297+
})
298+
},
299+
})
300+
return { authorizations, server }
301+
}),
302+
({ authorizations, server }) =>
303+
Effect.gen(function* () {
304+
const modelState = yield* Model.Service
305+
const bus = yield* Bus.Service
306+
const providerID = Provider.ID.make("rotating")
307+
const origin = `${server.url.origin}/v1`
308+
const models = { "rotating-model": {} }
309+
const test = yield* addPlugins([configuration("rotating", origin, models, "one")])
310+
311+
yield* eventually(
312+
modelState.get(providerID, Model.ID.make("rotating-model")),
313+
(model) => model?.limit.context === 111_111,
314+
)
315+
yield* test.setEntries([configuration("rotating", origin, models, "two")])
316+
yield* bus.publish(Event.Updated, {})
317+
yield* eventually(
318+
modelState.get(providerID, Model.ID.make("rotating-model")),
319+
(model) => model?.limit.context === 222_222,
320+
)
321+
expect(authorizations).toContain("Bearer two")
322+
}),
323+
({ server }) => Effect.promise(() => server.stop(true)),
324+
),
325+
)
245326
})

‎packages/web/src/content/docs/providers.mdx‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2685,8 +2685,12 @@ Precedence, highest first:
26852685
26862686
Discovery fails open: if the endpoint is unreachable, errors, or returns
26872687
malformed data, OpenCode keeps whatever limits it already has and never blocks
2688-
startup. A manual `limit` in the config is still the right choice when the
2689-
server does not advertise limits or advertises wrong ones.
2688+
startup. Failed lookups retry about once a minute and successful ones refresh
2689+
every five minutes, so an endpoint that was down when OpenCode started catches
2690+
up on its own. Providers with their own discovery integration (LM Studio,
2691+
Ollama, vLLM) are left to that integration and are never probed here. A manual
2692+
`limit` in the config is still the right choice when the server does not
2693+
advertise limits or advertises wrong ones.
26902694
26912695
---
26922696

0 commit comments

Comments
 (0)