Skip to content

Commit 3c94b9a

Browse files
committed
fix(routing): check noVisionModels before deriving the image modality
Review feedback on #2100: the migration resolved `modelInputModalities` without first consulting `noVisionModels`, so the evidence still disagreed with the resolver it claims to describe. Given noVisionModels: ["gpt-oss"] modelInputModalities: { "gpt-oss:120b": ["text", "image"] } `isModelTextOnly` matches the no-vision list and returns true before it ever reads the modality map (src/vision/index.ts:32), so the runtime says text-only while `candidateCapabilityEvidence` reported `image: true`. This is worse here than on the CLI surface #2086 fixed: routing acts on this evidence, so it can select a candidate for image work that execution then refuses. Same ordering, same primitives as the merged #2086. Also strengthens the window oracle the review flagged: `not.toBe(8_000)` also passed for `undefined` and for any other wrong value, so it is now the exact expected number. Two new tests. The positive one is red without this change; the negative one (a model outside noVisionModels keeps its declared image modality) passes either way on purpose -- it guards the fix from over-reaching rather than demonstrating the defect. Against origin/dev the file is 5 fail / 3 pass; against this branch's previous commit, 1 fail / 7 pass.
1 parent b01ed5f commit 3c94b9a

2 files changed

Lines changed: 54 additions & 8 deletions

File tree

‎src/routing/capability.ts‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
* how that affects eligibility.
1111
*/
1212

13-
import type { OcxConfig } from "../types";
13+
import { modelInList, type OcxConfig } from "../types";
1414
import { isCanonicalOpenAiForwardProvider, OPENAI_CODEX_PROVIDER_ID } from "../providers/openai-tiers";
1515
import { serviceTierSupportForModel } from "../providers/service-tier";
1616
import { PROVIDER_REGISTRY } from "../providers/registry";
@@ -176,10 +176,21 @@ export function candidateCapabilityEvidence(
176176
? (nativeOpenAiContextWindow(modelId, nativeContextLimits(config)) ?? rawContextWindow)
177177
: rawContextWindow;
178178

179-
const modalities = modelRecordValue(provider?.modelInputModalities, modelId)
180-
?? modelRecordValue(registryEntry?.modelInputModalities, modelId)
181-
?? catalogRow?.inputModalities
182-
?? (isNative ? nativeInputModalities(modelId) : undefined);
179+
// `noVisionModels` is checked before the modality chain because that is the order
180+
// `isModelTextOnly` uses: it matches the no-vision list and returns true before it
181+
// ever reads `modelInputModalities` (`src/vision/index.ts:32`). So a `gpt-oss`
182+
// no-vision entry beats an exact `gpt-oss:120b` entry that lists "image", and
183+
// deriving `image` from the modality chain alone reported vision on a model the
184+
// runtime refuses it for. That matters more here than on the CLI surface fixed in
185+
// #2086: routing *acts* on this evidence, so it would select the candidate for image
186+
// work that execution then rejects.
187+
const noVision = modelInList(provider?.noVisionModels, modelId);
188+
const modalities = noVision
189+
? ["text"]
190+
: (modelRecordValue(provider?.modelInputModalities, modelId)
191+
?? modelRecordValue(registryEntry?.modelInputModalities, modelId)
192+
?? catalogRow?.inputModalities
193+
?? (isNative ? nativeInputModalities(modelId) : undefined));
183194
const image = Array.isArray(modalities)
184195
? modalities.includes("image")
185196
: undefined;

‎tests/routing-capability-model-matching.test.ts‎

Lines changed: 38 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,14 +49,16 @@ describe("candidateCapabilityEvidence model matching", () => {
4949
});
5050

5151
test("the window does not fall through to the provider-wide value", () => {
52-
// The specific regression: 8_000 here is not "unknown", it is a definite answer
53-
// belonging to a different model, and routing would act on it.
52+
// The specific regression: the provider-wide 8_000 is not "unknown", it is a
53+
// definite answer belonging to a different model, and routing would act on it.
54+
// Asserted as the exact expected number rather than `not.toBe(8_000)`, which
55+
// would also pass for `undefined` or any other wrong value.
5456
const evidence = candidateCapabilityEvidence(
5557
configFor(providerWithFamilyEntries()),
5658
"custom",
5759
"gpt-oss:120b",
5860
);
59-
expect(evidence.contextWindow).not.toBe(8_000);
61+
expect(evidence.contextWindow).toBe(131_072);
6062
});
6163

6264
test("an exact entry still wins over the family entry", () => {
@@ -100,6 +102,39 @@ describe("candidateCapabilityEvidence model matching", () => {
100102
expect(evidence.reasoningEfforts).toEqual(registryEntry.modelReasoningEfforts![family]);
101103
});
102104

105+
test("noVisionModels beats an exact modality entry, as isModelTextOnly does", () => {
106+
// `isModelTextOnly` matches the no-vision list and returns true before it ever
107+
// reads `modelInputModalities`, so the `gpt-oss` no-vision entry wins over an
108+
// exact `gpt-oss:120b` entry listing "image". Evidence that disagrees here is
109+
// worse than a wrong window: routing selects the candidate for image work and
110+
// execution then refuses it.
111+
const provider = {
112+
...providerWithFamilyEntries(),
113+
noVisionModels: ["gpt-oss"],
114+
modelInputModalities: { "gpt-oss:120b": ["text", "image"] },
115+
} as unknown as OcxProviderConfig;
116+
117+
// Ground truth first: the resolver this evidence claims to describe says text-only.
118+
expect(isModelTextOnly(provider, "gpt-oss:120b")).toBe(true);
119+
120+
const evidence = candidateCapabilityEvidence(configFor(provider), "custom", "gpt-oss:120b");
121+
expect(evidence.image).toBe(false);
122+
});
123+
124+
test("a model outside noVisionModels keeps its declared image modality", () => {
125+
// The negative half: the no-vision check must not spread to models the list does
126+
// not cover, or the fix would trade a false positive for a false negative.
127+
const provider = {
128+
...providerWithFamilyEntries(),
129+
models: ["gpt-oss:120b", "llava:13b"],
130+
noVisionModels: ["gpt-oss"],
131+
modelInputModalities: { "llava:13b": ["text", "image"] },
132+
} as unknown as OcxProviderConfig;
133+
134+
expect(isModelTextOnly(provider, "llava:13b")).toBe(false);
135+
expect(candidateCapabilityEvidence(configFor(provider), "custom", "llava:13b").image).toBe(true);
136+
});
137+
103138
test("a prototype-shaped model id resolves nothing", () => {
104139
// modelRecordValue uses hasOwnProperty; a bare lookup would return Object.prototype
105140
// members here and hand routing a function as evidence.

0 commit comments

Comments
 (0)