Skip to content

Commit 1c82ec8

Browse files
committed
fix(routing): resolve capability evidence the way the resolver resolves it
candidateCapabilityEvidence read modelContextWindows, modelInputModalities and modelReasoningEfforts with bare lookups, while every runtime reader of those maps goes through modelRecordValue, which accepts a family entry for a tagged id. With contextWindow 8_000, modelContextWindows {"gpt-oss": 131_072}, modelInputModalities {"gpt-oss": ["text"]} and modelReasoningEfforts {"gpt-oss": ["low","high"]}, for gpt-oss:120b: runtime 131_072, text-only, [low, high] evidence {"contextWindow":8000,"tools":true,...} The window is the worst of the three. It did not degrade to unknown -- it fell through to the provider-wide contextWindow, so routing acted on a definite value belonging to a different model. That is exactly what this module's "unknown is not zero" contract exists to prevent. The other two dimensions simply went missing, which at least reads as unknown. A bare lookup also answered for prototype-shaped ids: a model named "constructor" resolved Object.prototype.constructor as its evidence. modelRecordValue uses hasOwnProperty, so that now resolves nothing. Five tests; three are red without the src change, two are guards against the fix over-reaching and pass either way. 68 tests green across the six routing/capability files. tsc --noEmit clean.
1 parent abaa75a commit 1c82ec8

2 files changed

Lines changed: 106 additions & 6 deletions

File tree

‎src/routing/capability.ts‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import {
2121
nativeReasoningEfforts,
2222
} from "../codex/catalog/metadata";
2323
import { readCatalog, readCodexCatalogPath } from "../codex/catalog/parsing";
24+
import { modelRecordValue } from "../reasoning-effort";
2425
import { statSync } from "node:fs";
2526
import type { RouteCapabilityEvidence } from "./trace";
2627

@@ -159,9 +160,14 @@ export function candidateCapabilityEvidence(
159160
const catalogRow = cachedCatalogModels().find(model => model.provider === providerName && model.id === modelId);
160161
const isNative = providerName === OPENAI_CODEX_PROVIDER_ID && !modelId.includes("/");
161162

162-
const rawContextWindow = provider?.modelContextWindows?.[modelId]
163+
// `modelRecordValue`, not a bare lookup: every runtime reader of these three maps
164+
// resolves them that way, so a `gpt-oss` entry covers `gpt-oss:120b`. Reading raw
165+
// made the evidence disagree with the resolver it claims to describe — and for the
166+
// window it did not even degrade to unknown, it fell through to the provider-wide
167+
// value, which is a definite wrong answer rather than an absent one.
168+
const rawContextWindow = modelRecordValue(provider?.modelContextWindows, modelId)
163169
?? provider?.contextWindow
164-
?? registryEntry?.modelContextWindows?.[modelId]
170+
?? modelRecordValue(registryEntry?.modelContextWindows, modelId)
165171
?? catalogRow?.contextWindow
166172
?? (isNative ? nativeOpenAiContextWindow(modelId, nativeContextLimits(config)) : undefined);
167173
// Native rows go through the accessor (raise-to-ceiling + opt-in). Routed rows keep
@@ -170,8 +176,8 @@ export function candidateCapabilityEvidence(
170176
? (nativeOpenAiContextWindow(modelId, nativeContextLimits(config)) ?? rawContextWindow)
171177
: rawContextWindow;
172178

173-
const modalities = provider?.modelInputModalities?.[modelId]
174-
?? registryEntry?.modelInputModalities?.[modelId]
179+
const modalities = modelRecordValue(provider?.modelInputModalities, modelId)
180+
?? modelRecordValue(registryEntry?.modelInputModalities, modelId)
175181
?? catalogRow?.inputModalities
176182
?? (isNative ? nativeInputModalities(modelId) : undefined);
177183
const image = Array.isArray(modalities)
@@ -196,8 +202,8 @@ export function candidateCapabilityEvidence(
196202
|| provider?.parallelToolCalls === true
197203
|| undefined;
198204

199-
const reasoningEfforts = provider?.modelReasoningEfforts?.[modelId]
200-
?? registryEntry?.modelReasoningEfforts?.[modelId]
205+
const reasoningEfforts = modelRecordValue(provider?.modelReasoningEfforts, modelId)
206+
?? modelRecordValue(registryEntry?.modelReasoningEfforts, modelId)
201207
?? (isNative ? nativeReasoningEfforts(modelId) : undefined);
202208

203209
const tierSupport = provider
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
import { describe, expect, test } from "bun:test";
2+
import { candidateCapabilityEvidence } from "../src/routing/capability";
3+
import { modelRecordValue } from "../src/reasoning-effort";
4+
import { isModelTextOnly } from "../src/vision";
5+
import type { OcxConfig, OcxProviderConfig } from "../src/types";
6+
7+
/**
8+
* `candidateCapabilityEvidence` describes what the resolver will do with a candidate,
9+
* so it has to match the resolver. Every runtime reader of `modelContextWindows`,
10+
* `modelInputModalities` and `modelReasoningEfforts` goes through `modelRecordValue`
11+
* (`src/reasoning-effort.ts:108`, `src/server/effort-policy.ts:122`,
12+
* `src/vision/index.ts:34`, `src/codex/catalog/provider-fetch.ts:612`), which accepts a
13+
* family entry for a tagged id. This file pins the evidence to that same rule.
14+
*
15+
* The window matters most: a bare lookup did not degrade to unknown there, it fell
16+
* through to the provider-wide `contextWindow` — a definite wrong answer, which the
17+
* module's own "unknown is not zero" contract is written to avoid.
18+
*/
19+
20+
function providerWithFamilyEntries(): OcxProviderConfig {
21+
return {
22+
adapter: "openai-chat",
23+
baseUrl: "https://example.test/v1",
24+
contextWindow: 8_000,
25+
models: ["gpt-oss:120b"],
26+
modelContextWindows: { "gpt-oss": 131_072 },
27+
modelInputModalities: { "gpt-oss": ["text"] },
28+
modelReasoningEfforts: { "gpt-oss": ["low", "high"] },
29+
} as unknown as OcxProviderConfig;
30+
}
31+
32+
function configFor(provider: OcxProviderConfig): OcxConfig {
33+
return { providers: { custom: provider } } as unknown as OcxConfig;
34+
}
35+
36+
describe("candidateCapabilityEvidence model matching", () => {
37+
test("a family entry covers its tagged siblings, as the resolver does", () => {
38+
const provider = providerWithFamilyEntries();
39+
40+
// Ground truth first: what the runtime itself resolves off this config.
41+
expect(modelRecordValue(provider.modelContextWindows, "gpt-oss:120b")).toBe(131_072);
42+
expect(isModelTextOnly(provider, "gpt-oss:120b")).toBe(true);
43+
44+
const evidence = candidateCapabilityEvidence(configFor(provider), "custom", "gpt-oss:120b");
45+
expect(evidence.contextWindow).toBe(131_072);
46+
expect(evidence.image).toBe(false);
47+
expect(evidence.reasoningEfforts).toEqual(["low", "high"]);
48+
});
49+
50+
test("the window does not fall through to the provider-wide value", () => {
51+
// The specific regression: 8_000 here is not "unknown", it is a definite answer
52+
// belonging to a different model, and routing would act on it.
53+
const evidence = candidateCapabilityEvidence(
54+
configFor(providerWithFamilyEntries()),
55+
"custom",
56+
"gpt-oss:120b",
57+
);
58+
expect(evidence.contextWindow).not.toBe(8_000);
59+
});
60+
61+
test("an exact entry still wins over the family entry", () => {
62+
const provider = {
63+
...providerWithFamilyEntries(),
64+
modelContextWindows: { "gpt-oss": 131_072, "gpt-oss:20b": 32_000 },
65+
} as unknown as OcxProviderConfig;
66+
expect(candidateCapabilityEvidence(configFor(provider), "custom", "gpt-oss:20b").contextWindow)
67+
.toBe(32_000);
68+
});
69+
70+
test("an unrelated model still falls back to the provider-wide window", () => {
71+
const evidence = candidateCapabilityEvidence(
72+
configFor(providerWithFamilyEntries()),
73+
"custom",
74+
"some-other-model",
75+
);
76+
expect(evidence.contextWindow).toBe(8_000);
77+
expect(evidence.reasoningEfforts).toBeUndefined();
78+
});
79+
80+
test("a prototype-shaped model id resolves nothing", () => {
81+
// modelRecordValue uses hasOwnProperty; a bare lookup would return Object.prototype
82+
// members here and hand routing a function as evidence.
83+
for (const modelId of ["constructor", "toString", "valueOf", "hasOwnProperty"]) {
84+
const evidence = candidateCapabilityEvidence(
85+
configFor(providerWithFamilyEntries()),
86+
"custom",
87+
modelId,
88+
);
89+
expect(evidence.contextWindow).toBe(8_000);
90+
expect(evidence.reasoningEfforts).toBeUndefined();
91+
expect(evidence.image).toBeUndefined();
92+
}
93+
});
94+
});

0 commit comments

Comments
 (0)