Skip to content

Commit 6a6efa9

Browse files
lidge-junrrmlimaCommandCodeBotjun
authored
feat(catalog): per-provider retainModels opt-in (#1690, carries #2860) (#3206)
* feat(catalog): honor per-provider retainModels opt-in (closes #1690) Operators can pin a model id so live discovery keeps it even when the upstream catalog omits it. The new opt-in is purely additive: built-in kimi/xai tables and the Vertex default continue to win, unknown ids are never inflated, and the existing one-line diagnostic still flags ids the live catalog dropped. Co-authored-by: CommandCodeBot <[email protected]> * feat(catalog): make retainModels self-sufficient and configurable (#1690) Builds on the carried #2860 predicate. retainModels ids now enter the configured seed (ordered union with the Vertex default and models), so a retain-only id survives live discovery, liveModels:false, and gets the same provider hints; selectedModels precedence is unchanged. Adds zod schema and superRefine validation, management PATCH/DTO/safe-config plumbing, ocx provider edit --retain-models <ids|->, rename-migration coverage, discovery-flight fingerprint, and docs. Design input: #2122 (union, schema, MODEL_ID_LISTS). --------- Co-authored-by: rrmlima <[email protected]> Co-authored-by: CommandCodeBot <[email protected]> Co-authored-by: jun <[email protected]>
1 parent 53c09a2 commit 6a6efa9

14 files changed

Lines changed: 505 additions & 3 deletions

File tree

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# wp4 — #1690 retainModels allowlist (two rival PRs)
2+
3+
Issue #1690 (score 58, labels enhancement/catalog). Two open drafts implement it:
4+
5+
| | PR #2860 (rrmlima) | PR #2122 (chilung-cgu) |
6+
| --- | --- | --- |
7+
| size | +132/-2, 3 files | +726/-25, 15 files |
8+
| CI at head | fully green (test 1-4, macos, ci) | only hygiene/label/target ran |
9+
| base | e546c160b, 310 behind dev; cherry-picks cleanly onto `fcf0da257` | same base |
10+
| retention | `shouldRetainConfiguredProviderModel(name, id, prov)` + `modelInList` | new `providerRetainModels` set inside the merge loop |
11+
| ids must also be in `models`? | yes (purely retentive) | no (`retainModels` folded into `configuredIds`) |
12+
| config validation | none (relies on `.passthrough()`) | zod `retainModels` + `nonBlankStringArrayConfigError` |
13+
| management API / DTO | none | none (neither exposes it via PATCH) |
14+
| rename migration | none | adds `retainModels` to `MODEL_ID_LISTS` |
15+
| 404 diagnostic | none | new module state + `warnRetainedModel404Once` wired into 4 request handlers, `CatalogModel.retainedWithoutDiscovery` |
16+
| docs | none | 5 locales, one table row each |
17+
18+
## Decision
19+
20+
Adopt **#2860 as the base commit** (cherry-picked as `2be9d505d` on
21+
`codex/retain-models-1690`), then add the pieces that make the opt-in
22+
discoverable and safe to hand-edit. #2122 is closed as superseded with credit for
23+
the config/migration design.
24+
25+
Why #2860 over #2122: the retention decision belongs in the one predicate the
26+
merge loop already consults; #2122 adds a second set beside it. #2122's 404
27+
diagnostic requires module-level maps keyed by provider, a new `CatalogModel`
28+
field, and edits to `core.ts`, `chat-completions.ts`, `chat-native.ts`,
29+
`claude-messages.ts` — four hot request paths — to print one warning that the
30+
upstream error body already carries (`model_not_found`). That is the wrong
31+
trade for this cycle; the existing `warnDroppedConfiguredIdsOnce` stays as the
32+
diagnostic for ids that are *not* retained.
33+
34+
## What this cycle adds on top of #2860
35+
36+
1. **`retainModels` alone is enough.** `configuredIds` in
37+
`fetchProviderModelsWithAuth` becomes the ordered union of the Vertex seed,
38+
`prov.models`, and `prov.retainModels`. An operator who writes
39+
`"retainModels": ["gemini-3.7-flash"]` should not have to repeat the id in
40+
`models`; requiring both is the footgun #2122 correctly avoided. #2860's
41+
"does not invent ids" test flips to assert the union.
42+
2. **Schema + load normalization** (`src/config.ts`): `retainModels:
43+
z.array(z.string().min(1)).transform(normalizeNonBlankStringArray).optional()`
44+
next to `noStructuredOutputModels`, plus the same `superRefine` entry so a
45+
hand-edited `"retainModels": "x"` fails with a path instead of being silently
46+
passed through.
47+
3. **Management PATCH + DTO** (`src/server/management/provider-routes.ts`):
48+
`retainModels` accepted like `noStructuredOutputModels` (`null` clears,
49+
empty array clears, validated with `nonBlankStringArrayConfigError`), and
50+
returned in the safe provider DTO so the dashboard/API round-trips it.
51+
4. **CLI opt-in** (`src/cli/provider-runtime.ts`):
52+
`ocx provider edit <name> --retain-models <id,id|->`. This is the easy
53+
switch: one flag, no JSON editing, `-` clears. Usage string and
54+
`skills/ocx` surface are regenerated if the capability registry changes
55+
(it does not — `provider edit` already exists; only the flag list grows).
56+
5. **Rename migration** (`src/providers/model-rename-migration.ts`):
57+
`"retainModels"` added to `MODEL_ID_LISTS` so a retired id is renamed
58+
rather than resurrected as a ghost row.
59+
6. **Docs** (`docs-site/.../reference/configuration/providers.md`): one table
60+
row after `selectedModels`, and a short paragraph in "Static model
61+
allowlists" contrasting `selectedModels` (narrows) with `retainModels`
62+
(preserves). English only this cycle; locales already lag on
63+
`noStructuredOutputModels` and a missing row does not contradict.
64+
65+
## Explicitly not in this cycle
66+
67+
- Seeding `CALLABLE_CONFIGURED_COMPATIBILITY_MODELS` with antigravity
68+
`gemini-3.7-flash` (issue step 4). That is a product default, and #1683 is a
69+
separate issue; with the config key available it no longer needs a release.
70+
- GUI field. The dashboard provider editor is untouched; the PATCH contract is
71+
ready for it, and a later PR can add the input with its screenshot.
72+
- 404-time warning. See "Decision".
73+
74+
## Acceptance criteria
75+
76+
- `retainModels` absent/empty → catalog identical to today (existing
77+
`tests/codex-catalog.test.ts` retention tests untouched and green).
78+
- `retainModels: ["x"]`, live omits `x`, `x` not in `models` → `x` present
79+
with provider hints applied; `droppedConfiguredIds` excludes it.
80+
- `retainModels: ["x"]`, live returns `x` → single row, no duplicate.
81+
- `liveModels: false` → `retainModels` ids are part of the static list.
82+
- Config load rejects `retainModels: "x"` / `[""]` with a
83+
`providers.<name>.retainModels` path; trims and dedupes valid input.
84+
- Management PATCH sets/clears; DTO echoes; CLI flag round-trips through PATCH.
85+
- A test through `fetchProviderModels` (not only `mergeConfiguredModelsIntoLiveCatalog`) proves a retain-only id survives both live discovery and `liveModels: false` (audit r1 blocker 2).
86+
- CLI treats `-` before `csv` so `--retain-models -` clears (audit r1 blocker 3).
87+
- `providerCatalogFingerprint` includes `retainModels`.
88+
- Migration renames a retired id inside `retainModels`.
89+
- `bun x tsc --noEmit` clean, `bun run privacy:scan` clean, focused files:
90+
`tests/catalog-retain-models.test.ts`, `tests/codex-catalog.test.ts`,
91+
`tests/management-provider-validation.test.ts`,
92+
`tests/model-rename-migration.test.ts` (if present), provider-runtime CLI test.
93+
94+
## Files
95+
96+
- `src/codex/catalog/provider-fetch.ts` — configuredIds union (on top of #2860).
97+
- `src/config.ts` — schema + superRefine.
98+
- `src/server/management/provider-routes.ts` — PATCH + DTO.
99+
- `src/server/auth-cors.ts` — `providerManagementConfigError` validation + safe-config DTO key list (audit r1 blocker 1).
100+
- `src/cli/provider-runtime.ts` — `--retain-models`.
101+
- `src/providers/model-rename-migration.ts` — list entry.
102+
- `src/types/provider.ts` — already added by #2860 (doc comment adjusted for union).
103+
- `docs-site/src/content/docs/reference/configuration/providers.md`.
104+
- `tests/catalog-retain-models.test.ts` (extend), `tests/management-provider-validation.test.ts` (extend).
105+
106+
## Closure
107+
108+
PR targets `dev`, `Closes #1690`, description names #2860 as the carried
109+
source commit (`12e69c200`) and #2122 as design input. After landing: close
110+
#1690 with the landing SHA, close #2860 as landed-via-carry (author credited in
111+
the squash trailer), close #2122 as superseded with the reasoning above.
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# wp4 audit r1 — synthesis
2+
3+
Reviewer: grok-4.6 subagent (Feynman), read-only, 7 questions on 040. Verdict: **near-pass / GO-WITH-FIXES** (3 blockers, 3 suggestions).
4+
5+
## Answers that confirm the plan
6+
7+
- Union (Q1): safe only if written as an ordered dedupe set `[vertexDefault?, ...models, ...retainModels]`, replacing the current `seed ? [default] : models` ternary. `configured` is the single seed for static `liveModels:false`, the Cursor filter, the degraded fallback, `droppedConfiguredIds`, and hints, so a retain-only id gets the same context/effort maps as a `models[]` entry. The Vertex seed predicate (`models.length === 0`) stays untouched.
8+
- Family match (Q2): acceptable, same semantics as `noVisionModels`. `retainModels: ["gpt-oss"]` keeps `gpt-oss:120b` and also invents a bare `gpt-oss` row through the union — extra row, never a drop.
9+
- selectedModels precedence (Q3): `filterCatalogVisibleModels` and `sync.ts` still hide a retained id when `selectedModels` is non-empty and omits it. Keep that; document "retain ≠ visible".
10+
- PATCH (Q4): copy `provider-routes.ts:386` verbatim; also GET list (:540).
11+
- CLI (Q5): no `takeListOption`; use `csv(takeOption)` and special-case `"-"` **before** `csv` (`csv("-")` yields `["-"]`).
12+
- #2122 (Q6): union, schema, `MODEL_ID_LISTS` are the necessary parts; the 404 module maps and `retainedWithoutDiscovery` are not. The `withConfiguredRetention(live→forCache)` change is incidental — the double merge is the OCX-111 combo-cache contract. Do not copy.
13+
- No-regression (Q7): absent/empty is a no-op; kimi/xai tables unchanged.
14+
15+
## Blockers folded into 040
16+
17+
1. `src/server/auth-cors.ts` was missing from the file list: `providerManagementConfigError` (~693) must validate `retainModels` with `nonBlankStringArrayConfigError`, and the safe-config DTO key list (~794) must include it, otherwise PATCH validation and DTO echo silently miss.
18+
2. The "liveModels:false / retain-only present" criterion cannot be proven through `mergeConfiguredModelsIntoLiveCatalog` alone. Add a test that goes through `fetchProviderModels`/the gather path so the union at :1307 is actually exercised.
19+
3. CLI: handle `-` before `csv`.
20+
21+
## Suggestions taken
22+
23+
- Docs state that `selectedModels` still narrows what is visible even for retained ids.
24+
- `retainModels` added to `providerCatalogFingerprint` (:573) so two providers differing only in that list do not share a discovery flight.
25+
- Flip #2860's "does not invent ids" test to assert the union.
26+
27+
## Disposition
28+
29+
All three blockers are additive edits inside the already-planned files plus one file (`auth-cors.ts`). No scope change. Proceed to B.
30+

‎docs-site/src/content/docs/reference/configuration/providers.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ differing backup and rewrites known legacy namespaced selected ids to bare ids.
8383
| `models?` | `string[]` | Seed/fallback model list. With `liveModels: false`, these are the only discovered models. |
8484
| `liveModels?` | `boolean` | Fetch the live catalog on start/sync (default `true`). Custom providers use `${baseUrl}/models`; built-ins may use a registry URL and filter. |
8585
| `selectedModels?` | `string[]` | Catalog allowlist after discovery. Non-empty exposes only those ids; empty or omitted exposes all discovered models. |
86+
| `retainModels?` | `string[]` | Ids kept in the catalog even when live discovery omits them. They need not be repeated in `models`. Empty or omitted keeps today's behavior. |
8687
| `contextWindow?` | `number` | Provider-wide context fallback when upstream metadata is absent; otherwise a cap that retains smaller live metadata. The Models dashboard exposes this separately from `providerContextCaps`. |
8788
| `modelContextWindows?` | `Record<string, number>` | Per-model context fallbacks/caps. These override `contextWindow`: an unknown window uses the configured value, while smaller live metadata remains authoritative. |
8889
| `modelInputModalities?` | `Record<string, string[]>` | Per-model input hints such as `["text"]` or `["text", "image"]`. |
@@ -575,6 +576,14 @@ silently replaced or truncated.
575576
Use `selectedModels` when discovery should still run but only selected ids should appear in Codex and
576577
`/v1/models`. The dashboard retains the full discovered list for later allowlist changes.
577578

579+
Use `retainModels` for the opposite problem: a provider whose `/models` endpoint omits an id that is
580+
still callable (a private deployment, a preview id, an OpenAI-compatible gateway with a partial
581+
listing). Listed ids are kept in the routed catalog with the same context and effort hints as
582+
`models`, and they survive `liveModels: false` too. `selectedModels` still narrows what is visible,
583+
so an id must be in both lists when an allowlist is active. Retaining an id does not make the
584+
upstream accept it; a wrong id fails at request time with the upstream error. From the CLI:
585+
`ocx provider edit <name> --retain-models gemini-3.7-flash,other-id` (`-` clears).
586+
578587
Preview GPT-5.6 fallback entries use the same mechanism. The OpenAI API-key preset seeds base and Pro
579588
ids with context `922000` and max input `922000`; OpenRouter seeds `openai/gpt-5.6-sol`,
580589
`openai/gpt-5.6-terra`, and `openai/gpt-5.6-luna` with context `922000`. Pool/Direct advertises

‎src/cli/provider-runtime.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ const USAGE = `Usage:
2424
[--auth-mode <key|forward|oauth|local|->] [--note <text|->]
2525
[--api-key-transport <x-api-key|bearer|->]
2626
[--headers <json>] [--enabled <on|off>] [--live-models <on|off>]
27+
[--retain-models <id,id|->]
2728
[--allow-private-network <on|off>] [--json]
2829
ocx provider test <name> [--json]
2930
ocx provider quota [--refresh] [--json]
@@ -48,6 +49,7 @@ async function edit(argv: string[], deps: RuntimeApiDeps): Promise<void> {
4849
const note = cleared(takeOption(args, "--note"));
4950
const apiKeyTransport = cleared(takeOption(args, "--api-key-transport"));
5051
const headers = takeOption(args, "--headers");
52+
const retainModelsRaw = takeOption(args, "--retain-models");
5153
const enabled = takeBooleanOption(args, "--enabled");
5254
const liveModels = takeBooleanOption(args, "--live-models");
5355
const allowPrivateNetwork = takeBooleanOption(args, "--allow-private-network");
@@ -74,6 +76,10 @@ async function edit(argv: string[], deps: RuntimeApiDeps): Promise<void> {
7476
}
7577
}
7678
if (enabled !== undefined) patch.disabled = !enabled;
79+
if (retainModelsRaw !== undefined) {
80+
// `-` clears, matching the other `edit` scalars; test before csv() or it becomes ["-"].
81+
patch.retainModels = retainModelsRaw.trim() === "-" ? null : csv(retainModelsRaw);
82+
}
7783
if (liveModels !== undefined) patch.liveModels = liveModels;
7884
if (allowPrivateNetwork !== undefined) patch.allowPrivateNetwork = allowPrivateNetwork;
7985
if (Object.keys(patch).length === 0) throw new CliUsageError("at least one edit option is required", USAGE);

‎src/codex/catalog/provider-fetch.ts‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -571,6 +571,7 @@ function providerCatalogFingerprint(name: string, prov: OcxProviderConfig): Reco
571571
base: prov.baseUrl ?? "",
572572
adapter: prov.adapter ?? "",
573573
models: [...(prov.models ?? [])].sort(),
574+
retain: [...(prov.retainModels ?? [])].sort(),
574575
selected: [...(prov.selectedModels ?? [])].sort(),
575576
defaultModel: prov.defaultModel ?? null,
576577
ctx: prov.contextWindow ?? null,
@@ -1304,7 +1305,14 @@ async function fetchProviderModelsWithAuth(
13041305
&& prov.googleMode === "vertex"
13051306
&& (prov.models?.length ?? 0) === 0
13061307
&& Boolean(prov.defaultModel);
1307-
const configuredIds = seedVertexDefault && prov.defaultModel ? [prov.defaultModel] : (prov.models ?? []);
1308+
// Ordered dedupe union: Vertex seed, then `models`, then `retainModels`. `configured` is the
1309+
// single seed for the static path, the degraded fallback, drop diagnostics, and provider hints,
1310+
// so a retain-only id must enter here or it never exists to be retained (#1690).
1311+
const configuredIds = [...new Set([
1312+
...(seedVertexDefault && prov.defaultModel ? [prov.defaultModel] : []),
1313+
...(prov.models ?? []),
1314+
...(prov.retainModels ?? []),
1315+
])];
13081316
const configured: CatalogModel[] = configuredIds.map(id => ({
13091317
id,
13101318
provider: name,
@@ -1700,9 +1708,14 @@ export function shouldExposeProviderModel(providerName: string, modelId: string)
17001708
return true;
17011709
}
17021710

1703-
export function shouldRetainConfiguredProviderModel(providerName: string, modelId: string): boolean {
1711+
export function shouldRetainConfiguredProviderModel(
1712+
providerName: string,
1713+
modelId: string,
1714+
prov?: OcxProviderConfig,
1715+
): boolean {
17041716
if (CALLABLE_CONFIGURED_COMPATIBILITY_MODELS[providerName]?.has(modelId)) return true;
17051717
if (providerName === "opencode-free") return modelId === "big-pickle" || modelId.endsWith("-free");
1718+
if (modelInList(prov?.retainModels, modelId)) return true;
17061719
return false;
17071720
}
17081721

@@ -1748,7 +1761,7 @@ export function mergeConfiguredModelsIntoLiveCatalog(opts: {
17481761
}
17491762
if (
17501763
seedVertexDefault === true
1751-
|| shouldRetainConfiguredProviderModel(name, candidate.id)
1764+
|| shouldRetainConfiguredProviderModel(name, candidate.id, prov)
17521765
|| (retainComboTargets && retainConfiguredModelIds?.has(candidate.id) === true)
17531766
) {
17541767
out.push(candidate);

‎src/config.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -529,6 +529,9 @@ const providerConfigSchema = z.object({
529529
noStructuredOutputModels: z.array(z.string().min(1))
530530
.transform(normalizeNonBlankStringArray)
531531
.optional(),
532+
retainModels: z.array(z.string().min(1))
533+
.transform(normalizeNonBlankStringArray)
534+
.optional(),
532535
omitReasoningEffortWithToolsModels: z.array(z.string().min(1))
533536
.transform(normalizeNonBlankStringArray)
534537
.optional(),
@@ -1368,6 +1371,17 @@ const configSchema = z.object({
13681371
message: structuredOutputOptOutError,
13691372
});
13701373
}
1374+
const retainModelsError = nonBlankStringArrayConfigError(
1375+
(provider as { retainModels?: unknown }).retainModels,
1376+
"retainModels",
1377+
);
1378+
if (retainModelsError) {
1379+
ctx.addIssue({
1380+
code: "custom",
1381+
path: ["providers", redactSecretString(name), "retainModels"],
1382+
message: retainModelsError,
1383+
});
1384+
}
13711385
const toolReasoningOptOutError = nonBlankStringArrayConfigError(
13721386
(provider as { omitReasoningEffortWithToolsModels?: unknown }).omitReasoningEffortWithToolsModels,
13731387
"omitReasoningEffortWithToolsModels",

‎src/providers/model-rename-migration.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,9 @@ const MODEL_ID_LISTS = [
9999
// their catalog instead of being renamed. OAuth reconciliation does not cover this
100100
// field, so the rename has to.
101101
"selectedModels",
102+
// Same reasoning as `selectedModels`: a retired id pinned here would be resurrected as a
103+
// ghost row on every discovery instead of following the rename (#1690).
104+
"retainModels",
102105
"noVisionModels",
103106
"noReasoningModels",
104107
"noTemperatureModels",

‎src/server/auth-cors.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -694,6 +694,8 @@ export function providerManagementConfigError(name: unknown, provider: unknown):
694694
"noStructuredOutputModels",
695695
);
696696
if (structuredOutputOptOutError) return `provider ${name} ${structuredOutputOptOutError}`;
697+
const retainModelsError = nonBlankStringArrayConfigError(raw.retainModels, "retainModels");
698+
if (retainModelsError) return `provider ${name} ${retainModelsError}`;
697699
const toolReasoningOptOutError = nonBlankStringArrayConfigError(
698700
raw.omitReasoningEffortWithToolsModels,
699701
"omitReasoningEffortWithToolsModels",
@@ -792,6 +794,7 @@ export function safeConfigDTO(config: OcxConfig): unknown {
792794
"noTopPModels",
793795
"noPenaltyModels",
794796
"noStructuredOutputModels",
797+
"retainModels",
795798
"omitReasoningEffortWithToolsModels",
796799
"upstreamHttpVersion",
797800
"autoToolChoiceOnlyModels",

‎src/server/management/provider-routes.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -396,6 +396,19 @@ function applyProviderPatchFields(
396396
}
397397
touched = true;
398398
}
399+
if (Object.hasOwn(rawBody, "retainModels")) {
400+
const value = rawBody.retainModels;
401+
if (value === null) {
402+
delete next.retainModels;
403+
} else {
404+
const error = nonBlankStringArrayConfigError(value, "retainModels");
405+
if (error) return { error };
406+
const models = normalizeNonBlankStringArray(value as string[]);
407+
if (models.length > 0) next.retainModels = models;
408+
else delete next.retainModels;
409+
}
410+
touched = true;
411+
}
399412
if (Object.hasOwn(rawBody, "omitReasoningEffortWithToolsModels")) {
400413
const value = rawBody.omitReasoningEffortWithToolsModels;
401414
if (value === null) {
@@ -538,6 +551,7 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise<Resp
538551
modelAutoCompactTokenLimits: p.modelAutoCompactTokenLimits,
539552
modelSupportsServiceTier: p.modelSupportsServiceTier,
540553
noStructuredOutputModels: p.noStructuredOutputModels,
554+
retainModels: p.retainModels,
541555
omitReasoningEffortWithToolsModels: p.omitReasoningEffortWithToolsModels,
542556
upstreamHttpVersion: p.upstreamHttpVersion,
543557
authMode: p.authMode,

0 commit comments

Comments
 (0)