Skip to content

Commit 99a2924

Browse files
ntdatt812claude
andcommitted
fix(lab): report the model gates the way the adapters actually match them
`resolveProductionBehaviorValues` is documented as "authoritative effective values emitted by the production route/model/adapter resolver", and its hash is the behavior fingerprint that keys Lab evidence. Ten of its rows are membership tests over the provider's `no*Models`-style lists, and they went through a local function includesModel(list, modelId) { return Array.isArray(list) && list.includes(modelId); } while every runtime gate those rows describe matches through `modelInList`, which also accepts a bare entry for a tagged id. ollama-cloud serves `gpt-oss:120b`, `qwen3-coder:480b`, `qwen3.5:397b` and `gemma4:31b`, and the same registry row writes the bare `gpt-oss` into noVisionModels -- the bare-prefix form is how these lists are meant to be written. So with `noTemperatureModels: ["gpt-oss"]` the adapter omitted temperature from the request for `gpt-oss:120b` while the report said `sampling.omitTemperature: false`. Same for omitTopP, omitPenalties, reasoning.budgetMode, reasoning.splitMode, reasoning.toggleMode, reasoning.supported, reasoning.replayMode's two flags, and tools.choiceRestrictions. The blast radius is confined to subjects that were being described wrongly. Fingerprints measured on the same config, before and after: gpt-oss:120b 54154e19bd2c8ee4 -> 45a73577c5c257f9 (was wrong) gpt-oss 45a73577c5c257f9 -> 45a73577c5c257f9 (unchanged) glm-5.3 54154e19bd2c8ee4 -> 54154e19bd2c8ee4 (unchanged) Note the first line against the third: a model whose sampling gates were applied hashed identically to one where they were not. It now hashes with `gpt-oss`, the id it actually behaves like. Recorded evidence for unaffected subjects keeps its fingerprint, so `resolverVersion` is left at 2 rather than invalidating every recorded subject globally -- say the word if you would rather draw a clean generation boundary and I will bump it. Tests assert the wire the adapter really builds first, then hold the report to that same wire, so the pair cannot drift apart silently. The five report cases fail on current dev. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 69650fa commit 99a2924

2 files changed

Lines changed: 85 additions & 1 deletion

File tree

‎src/routing/compatibility/behavior.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { modelInList } from "../../types";
12
import type { OcxConfig, OcxProviderConfig } from "../../types";
23
import { PROVIDER_REGISTRY } from "../../providers/registry";
34
import { fastPolicyForModel, serviceTierSupportForModel } from "../../providers/service-tier";
@@ -37,8 +38,17 @@ function behaviorRow(source: LabBehaviorSource, value: unknown) {
3738
return { source, value };
3839
}
3940

41+
/**
42+
* Membership for the provider's `no*Models`-style lists.
43+
*
44+
* Delegates to modelInList so the report matches the wire: every runtime gate these
45+
* rows describe (openai-chat's sampling/reasoning/tool-choice gates, reasoning-effort's
46+
* noReasoningModels) matches through modelInList, which also accepts a bare entry for a
47+
* tagged id. ollama-cloud serves `gpt-oss:120b` and lists the bare `gpt-oss`, so an
48+
* exact-only check here reported "temperature is sent" on a request that omits it.
49+
*/
4050
function includesModel(list: string[] | undefined, modelId: string): boolean {
41-
return Array.isArray(list) && list.includes(modelId);
51+
return modelInList(list, modelId);
4252
}
4353

4454
function modelValue<T>(map: Record<string, T> | undefined, modelId: string): T | undefined {
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
/**
2+
* The compatibility behavior report is documented as "authoritative effective values
3+
* emitted by the production route/model/adapter resolver", and its hash keys Lab
4+
* evidence. These cases hold it to that: for one routed model they assert the wire the
5+
* adapter actually builds, then assert the report describes that same wire.
6+
*/
7+
import { describe, expect, test } from "bun:test";
8+
import { resolveProductionBehaviorValues } from "../src/routing/compatibility/behavior";
9+
import { createOpenAIChatAdapter } from "../src/adapters/openai-chat";
10+
import type { OcxConfig, OcxParsedRequest, OcxProviderConfig } from "../src/types";
11+
12+
// ollama-cloud ships `gpt-oss:120b` verbatim (src/providers/registry.ts) and the same
13+
// registry row lists the bare `gpt-oss` in noVisionModels, i.e. the bare-prefix form is
14+
// the documented, intended way to write these lists.
15+
const MODEL = "gpt-oss:120b";
16+
17+
const effective: OcxProviderConfig = {
18+
adapter: "openai-chat",
19+
baseUrl: "https://ollama.com/v1",
20+
apiKey: "sk-test",
21+
authMode: "key",
22+
noTemperatureModels: ["gpt-oss"],
23+
noTopPModels: ["gpt-oss"],
24+
noPenaltyModels: ["gpt-oss"],
25+
thinkingBudgetModels: ["gpt-oss"],
26+
autoToolChoiceOnlyModels: ["gpt-oss"],
27+
};
28+
29+
const config = { providers: { "ollama-cloud": effective } } as unknown as OcxConfig;
30+
31+
const values = () =>
32+
resolveProductionBehaviorValues(config, "ollama-cloud", MODEL, effective, "salt")!;
33+
34+
function wire(): Record<string, unknown> {
35+
const parsed: OcxParsedRequest = {
36+
modelId: MODEL,
37+
context: { messages: [{ role: "user", content: "hi", timestamp: 0 }] },
38+
stream: false,
39+
options: { temperature: 0.5, topP: 0.9, presencePenalty: 0.2, frequencyPenalty: 0.2 },
40+
};
41+
return JSON.parse(createOpenAIChatAdapter(effective).buildRequest(parsed).body as string);
42+
}
43+
44+
describe("behavior report must agree with the wire the adapter actually builds", () => {
45+
test("adapter really does omit these for the :tag model (ground truth)", () => {
46+
const body = wire();
47+
expect(body.temperature).toBeUndefined();
48+
expect(body.top_p).toBeUndefined();
49+
expect(body.presence_penalty).toBeUndefined();
50+
expect(body.frequency_penalty).toBeUndefined();
51+
});
52+
53+
test("report agrees: sampling.omitTemperature", () => {
54+
expect(values()["sampling.omitTemperature"]!.value).toBe(true);
55+
});
56+
test("report agrees: sampling.omitTopP", () => {
57+
expect(values()["sampling.omitTopP"]!.value).toBe(true);
58+
});
59+
test("report agrees: sampling.omitPenalties", () => {
60+
expect(values()["sampling.omitPenalties"]!.value).toBe(true);
61+
});
62+
test("report agrees: reasoning.budgetMode", () => {
63+
expect(values()["reasoning.budgetMode"]!.value).toBe(true);
64+
});
65+
test("report agrees: tools.choiceRestrictions", () => {
66+
expect(values()["tools.choiceRestrictions"]!.value).toEqual(["auto"]);
67+
});
68+
69+
test("an unlisted model reports false (control)", () => {
70+
const v = resolveProductionBehaviorValues(config, "ollama-cloud", "glm-5.3", effective, "salt")!;
71+
expect(v["sampling.omitTemperature"]!.value).toBe(false);
72+
expect(v["reasoning.budgetMode"]!.value).toBe(false);
73+
});
74+
});

0 commit comments

Comments
 (0)