Skip to content

Commit c59a5a2

Browse files
anandgupta42claude
andauthored
fix: correct the ChatGPT-subscription model allowlist against the live backend (#1179)
* fix: correct the ChatGPT-subscription model allowlist against the live backend The OAuth model filter was built from the models.dev catalog rather than from what the Codex endpoint actually serves, so it was wrong in both directions. Verified every gpt-5.x id against `POST /backend-api/codex/responses` on a ChatGPT Pro credential and rebuilt the allowlist from the results. Removed (offered in the picker, rejected by the backend with HTTP 400): - `gpt-5.2` and `gpt-5.6`, both explicitly allowlisted - `gpt-5.3-codex`, admitted by the `modelId.includes("codex")` auto-allow Added (served by the backend, previously hidden from the picker): - `gpt-5.6-sol`, `gpt-5.6-luna`, `gpt-5.6-terra` — the current flagship subscription models, excluded on an untested assumption that the sol/luna/terra variants were API-tier-only Dropped the `includes("codex")` substring auto-allow. It cannot express the real policy: the tier accepts `gpt-5.3-codex-spark` but rejects `gpt-5.3-codex`, and accepts `gpt-5.6-sol` but rejects `gpt-5.6`. Membership is now exact-match only. Transport was never involved. Plain HTTP with our own `originator: altimate` identity reaches every working model; no websocket, `openai-beta`, or `x-codex-beta-features` header changes any outcome. Rewrote `codex-allowlist.test.ts` around the verified truth table so both failure directions are guarded, plus a regression barrier against reintroducing a substring rule. Closes #1178 Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * docs: record why the Codex model allowlist is static, not discovered The obvious objection to the allowlist this PR corrects is that any static list drifts. There is an authoritative per-account endpoint that would let us derive it instead — `GET /backend-api/codex/models?client_version=<v>` — so I probed it rather than leave the objection unanswered. It works, and it accepts our own identity: `originator: altimate` plus our own User-Agent, no impersonation. Its `visibility: "list"` slugs are exactly the seven ids in `OAUTH_ALLOWED_MODELS`, which independently confirms the table built by request-level probing. It is still unusable as a runtime source of truth. `client_version` is mandatory and is gated against each model's `minimal_client_version`, today 0.98.0 through 0.144.0 — Codex CLI release numbers, a line we are not on. Our versions sit below all of them and the gate fails silently: our published `0.9.7` returns `HTTP 200 {"models":[]}`, and a dev build's `local` returns `400 {"detail":"Invalid client_version format"}`. Deriving the list would mean asserting a Codex CLI version we are not, so the list stays static. Comment-only; no behavior change. Records the endpoint, the corroboration, the blocker, and the drop-in shape should a legitimate client_version ever exist, so the next person does not repeat the investigation. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix: match the OAuth model filter on api.id, not the config map key Addresses review on #1179. `shouldAllowOAuthModel` was called with the model map KEY. For every models.dev catalog entry the key equals `api.id`, so this was invisible — but a user can alias a model in config: provider.openai.models.fast-spark.id = "gpt-5.3-codex-spark" Config models are folded into the provider database ("extend database from config" in `Provider.state`) BEFORE auth loaders run, so that entry reaches this loader keyed `fast-spark` with the real id on `api.id`. Matching the key deleted it, hiding a model the backend actually serves. The sibling `plugin/openai/codex.ts` filter already matches on `model.api.id`. Extracted `disallowedOAuthModelKeys`, which resolves `api.id ?? key` — the fallback matters because the database only backfills `model.api.id ?? model.id ?? modelID` after this hook runs, so the field is not guaranteed populated despite the type. Behaviour is unchanged for every catalog model (key === api.id verified against models.dev/api.json); only aliases move, and an alias of a rejected id is still deleted. Comment corrections, no behaviour change: - The allowlist claimed every other gpt-5.x catalog id had been probed. It had not: gpt-5-mini, gpt-5-nano, gpt-5-pro and gpt-5.2-chat-latest are in the catalog and in neither list. Recorded as unprobed and fail-closed rather than left as an overclaim. - Recorded that the truth table is one Pro account's entitlements and has not been checked against a Plus credential. - Recorded that the codex-tagged ids reviewers keep asking about (gpt-5.1-codex, gpt-5.1-codex-max, gpt-5.1-codex-mini, gpt-5.2-codex, gpt-5.3-codex-xhigh) are absent from the models.dev catalog entirely, so no filter here can hide them. - Disambiguated the two version numbers. What reaches the wire is `Installation.VERSION` (the npm-published 0.9.7, or `local` for a dev build), not the 1.17.9 in packages/opencode/package.json, which is inherited from upstream and never sent. Also stated that the blocker is not arithmetic: bumping our number would not lift it, because the field asserts which Codex CLI release we are. Six new tests cover the alias fix in both directions, the missing-api.id fallback, and that the catalog set resolves identically either way. The allowlist membership is byte-for-byte unchanged, as is the `originator: "altimate"` identity. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * docs: correct a false catalog claim, and record the cold-cache caveat My previous commit asserted that gpt-5.1-codex, gpt-5.1-codex-max, gpt-5.1-codex-mini, gpt-5.2-codex and gpt-5.3-codex-xhigh "are not in the models.dev catalog at all", and concluded that no filter here could hide them. That was wrong, and the review that caught it was right. I had checked live https://models.dev/api.json and stopped there. There is a second catalog: the bundled provider/models-snapshot.ts. ModelsDev.Data resolves disk cache -> bundled snapshot -> fetch, so on a fresh install or cold cache the snapshot IS the catalog. Its `openai` provider still carries gpt-5-codex, gpt-5.1-codex, gpt-5.1-codex-max, gpt-5.1-codex-mini and gpt-5.2-codex. Those five did reach the loader, the removed includes("codex") rule did offer them, and exact matching does now delete them. None was individually probed. They stay excluded, on the same fail-closed reasoning applied to gpt-5.2: an unverified inclusion fails opaquely mid-request, an unverified exclusion fails visibly at selection. But that is a decision, and it is now recorded as one rather than dressed up as a discovery. The same staleness cuts the other way and is also now recorded: the bundled snapshot contains no gpt-5.6 variant at all, so on a cold cache sol/luna/terra are absent from the catalog and this allowlist cannot conjure them — the filter only deletes. The allowlist is necessary for them to appear but, on a cold cache, not sufficient; they surface once the catalog refreshes from models.dev or the snapshot is regenerated at the next release build. Adds a test pinning the five unprobed snapshot codex ids as excluded-by- decision, asserting they are in neither verified list, so a future probe has an obvious place to land. Comments and tests only. The allowlist membership is unchanged, as is request-time behaviour. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 6f6037f commit c59a5a2

2 files changed

Lines changed: 318 additions & 106 deletions

File tree

‎packages/opencode/src/plugin/codex.ts‎

Lines changed: 140 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -16,36 +16,152 @@ const CODEX_API_ENDPOINT = "https://chatgpt.com/backend-api/codex/responses"
1616
const OAUTH_PORT = 1455
1717
const OAUTH_POLLING_SAFETY_MARGIN_MS = 3000
1818

19-
/** Non-codex ChatGPT-subscription (OAuth) allowlist. Any modelId
20-
* containing "codex" is auto-allowed by ``shouldAllowOAuthModel`` below,
21-
* so this set only enumerates the plain non-codex main/mini variants
22-
* OpenAI exposes on Codex-tier accounts. Bump whenever a new gpt-5.N
23-
* is generally available on the subscription. Exported for unit-test
24-
* coverage — see test/plugin/codex-allowlist.test.ts. */
19+
/** Exact set of model ids the ChatGPT-subscription (Codex) tier accepts.
20+
*
21+
* Every entry was verified against the live backend
22+
* (POST https://chatgpt.com/backend-api/codex/responses) on a ChatGPT Pro
23+
* credential: entries here returned HTTP 200.
24+
*
25+
* CONFIRMED REJECTED, so deliberately absent — each returned
26+
* ``400 {"detail":"The '<id>' model is not supported when using Codex with a
27+
* ChatGPT account."}``: gpt-5, gpt-5.1, gpt-5.2, gpt-5.2-pro,
28+
* gpt-5.3-chat-latest, gpt-5.3-codex, gpt-5.4-nano, gpt-5.4-pro, gpt-5.5-pro,
29+
* gpt-5.6.
30+
*
31+
* NOT PROBED, and excluded by default because this list is fail-closed:
32+
* gpt-5-mini, gpt-5-nano, gpt-5-pro, gpt-5.2-chat-latest. They are in the
33+
* models.dev catalog but are not plausible Codex-tier models, and the
34+
* discovery endpoint below does not list them either. An earlier revision of
35+
* this comment claimed every other catalog id had been probed; that overstated
36+
* the evidence, and these four are the exception.
37+
*
38+
* TIER SCOPE. All of the above is what ONE ChatGPT Pro account was served.
39+
* It has not been verified against a Plus credential, so it is possible Plus
40+
* is entitled to a narrower (or wider) set. If a Plus subscriber reports a
41+
* model missing from the picker that the official client offers them, that is
42+
* the likely cause and the fix is a per-account list, not another id here.
43+
*
44+
* OLDER CODEX IDS ARE DROPPED, AND THAT DROP IS UNVERIFIED. Reviewers keep
45+
* asking why codex-tagged ids appear in neither list above. The honest answer
46+
* depends on WHICH catalog is live, because the two disagree:
47+
*
48+
* * live https://models.dev/api.json carries exactly two codex ids —
49+
* gpt-5.3-codex and gpt-5.3-codex-spark — both accounted for above.
50+
* * the BUNDLED snapshot (provider/models-snapshot.ts) is staler and still
51+
* carries gpt-5-codex, gpt-5.1-codex, gpt-5.1-codex-max,
52+
* gpt-5.1-codex-mini and gpt-5.2-codex under ``openai``.
53+
*
54+
* ModelsDev.Data resolves disk cache -> bundled snapshot -> fetch, so on a
55+
* fresh install or cold cache the snapshot IS the catalog, and those five ids
56+
* do reach this loader. The removed ``includes("codex")`` rule offered them;
57+
* exact matching now deletes them. None was individually probed. They are
58+
* absent from the discovery endpoint's nine slugs (a Pro account), which is
59+
* evidence but not a probe, and consistent with older codex models having been
60+
* retired.
61+
*
62+
* They stay out on the same fail-closed reasoning as the ids above: an
63+
* unverified inclusion fails opaquely mid-request, an unverified exclusion
64+
* fails visibly at selection with a "did you mean" list. Probe one and move it
65+
* into the right list to settle it.
66+
*
67+
* COLD-CACHE CAVEAT. That same snapshot staleness cuts the other way for the
68+
* models this list adds: it contains NO gpt-5.6 variant, so on a cold cache
69+
* sol/luna/terra are absent from the catalog entirely and this allowlist cannot
70+
* conjure them — the filter only ever deletes. They become selectable once the
71+
* catalog refreshes from models.dev (or the bundled snapshot is regenerated at
72+
* the next release build). This allowlist is necessary for them to appear, but
73+
* on a cold cache it is not sufficient.
74+
*
75+
* There is no derivable rule here — the tier accepts ``gpt-5.3-codex-spark``
76+
* but rejects ``gpt-5.3-codex``, and accepts the gpt-5.6 sol/luna/terra
77+
* variants but rejects plain ``gpt-5.6``. So this is an exact-match list by
78+
* necessity, not by preference. Only add an id after confirming a 200 from the
79+
* endpoint above on a subscription credential; guessing puts a model in the
80+
* picker that then fails at request time.
81+
*
82+
* WHY THIS IS STATIC AND NOT DISCOVERED. There is an authoritative per-account
83+
* endpoint — ``GET https://chatgpt.com/backend-api/codex/models?client_version=<v>``
84+
* — and it accepts our own identity (``originator: altimate`` plus our
85+
* User-Agent; no impersonation needed) and returns exactly the set above, each
86+
* entry carrying ``slug``, ``visibility`` and ``minimal_client_version``. Its
87+
* ``visibility: "list"`` slugs corroborate this list precisely, which is why
88+
* that list is reproduced here rather than derived at runtime:
89+
*
90+
* ``client_version`` is mandatory (omitted or unparseable ⇒ HTTP 400) and is
91+
* gated against each model's ``minimal_client_version``, which at the time of
92+
* probing ranged from 0.98.0 to 0.144.0. Those are Codex CLI release numbers —
93+
* a numbering line we are not on, so comparing our number against theirs is
94+
* meaningless regardless of which way it happens to sort.
95+
*
96+
* Two different version numbers exist in this repo and it matters which one
97+
* reaches the wire. What we would send is ``Installation.VERSION``, i.e. the
98+
* build-injected ``OPENCODE_VERSION`` — the NPM-published version, 0.9.7 at
99+
* the time of probing, or the literal ``local`` for a dev build. It is NOT the
100+
* 1.17.9 in packages/opencode/package.json, which is inherited from the
101+
* upstream numbering line and never sent. Both observed values fail, and the
102+
* failure is SILENT for one of them: ``client_version=0.9.7`` returns
103+
* ``HTTP 200 {"models":[]}`` — no error to debug, just an empty picker — while
104+
* a dev build's ``local`` returns
105+
* ``400 {"detail":"Invalid client_version format"}``.
106+
*
107+
* The blocker is not arithmetic, so bumping our version would not lift it: the
108+
* field means "which Codex CLI release am I", and answering it is impersonating
109+
* the first-party client whatever number we put there. We do not do that, so
110+
* the list stays static. If we ever have a legitimate client_version to send,
111+
* the mechanism is a small drop-in: fetch, keep ``visibility === "list"``, and
112+
* fall back to this set whenever the response is empty or the call fails.
113+
*
114+
* Exported for unit-test coverage — see test/plugin/codex-allowlist.test.ts. */
25115
export const OAUTH_ALLOWED_MODELS = new Set([
26-
"gpt-5.2",
116+
"gpt-5.3-codex-spark",
27117
"gpt-5.4",
28118
"gpt-5.4-mini",
29119
"gpt-5.5",
30-
"gpt-5.6",
120+
"gpt-5.6-luna",
121+
"gpt-5.6-sol",
122+
"gpt-5.6-terra",
31123
])
32124

33125
/** OAuth (ChatGPT-subscription) model-filter policy for the ACTIVE plugin
34-
* (this file — wired via plugin/index.ts). A model is kept if either
35-
* (a) its id contains ``"codex"`` (all codex variants ship on the
36-
* subscription), or (b) its id is an exact member of
37-
* ``OAUTH_ALLOWED_MODELS`` (the curated non-codex releases).
126+
* (this file — wired via plugin/index.ts). Exact membership in
127+
* ``OAUTH_ALLOWED_MODELS`` — no substring heuristics.
128+
*
129+
* This previously auto-allowed any id containing ``"codex"``, which admitted
130+
* ``gpt-5.3-codex``. The backend rejects that id, so it reached the picker and
131+
* then failed at request time with an HTTP 400. Substring matching cannot
132+
* express the real policy (``gpt-5.3-codex-spark`` is accepted while
133+
* ``gpt-5.3-codex`` is not), so the heuristic is gone.
38134
*
39-
* The sibling file plugin/openai/codex.ts (an in-progress refactor,
40-
* currently NOT wired) has its own separate filter with a
41-
* ``parseFloat(match[1]) > 5.4`` fallback. Adopting this helper is
42-
* followup work on that refactor — do NOT assume the two files share
43-
* this policy today. */
135+
* The sibling file plugin/openai/codex.ts (an in-progress refactor, currently
136+
* NOT wired) has its own separate filter with a ``parseFloat(match[1]) > 5.4``
137+
* fallback. Adopting this helper is followup work on that refactor — do NOT
138+
* assume the two files share this policy today. */
44139
export function shouldAllowOAuthModel(modelId: string): boolean {
45-
if (modelId.includes("codex")) return true
46140
return OAUTH_ALLOWED_MODELS.has(modelId)
47141
}
48142

143+
/** Map keys to delete from an OAuth (subscription) provider's model record.
144+
*
145+
* Matches on the UPSTREAM api id, not the map key. The two are equal for every
146+
* models.dev catalog entry, but a user can alias a model in config —
147+
* ``provider.openai.models.fast-spark.id = "gpt-5.3-codex-spark"`` — which
148+
* produces an entry keyed ``fast-spark`` whose ``api.id`` carries the real id.
149+
* Config models are folded into the provider database (Provider.state, "extend
150+
* database from config") BEFORE auth loaders run, so they do reach this filter;
151+
* matching the key would delete a model the backend actually serves.
152+
*
153+
* Falls back to the key when ``api.id`` is absent: the database only backfills
154+
* ``model.api.id ?? model.id ?? modelID`` after this hook has run, so the field
155+
* is not guaranteed populated at this point even though the type says so.
156+
*
157+
* Returns keys rather than mutating, so the caller keeps the in-place delete on
158+
* the shared database object and this stays unit-testable. */
159+
export function disallowedOAuthModelKeys(models: Record<string, { api?: { id?: string } }>): string[] {
160+
return Object.entries(models)
161+
.filter(([key, model]) => !shouldAllowOAuthModel(model?.api?.id ?? key))
162+
.map(([key]) => key)
163+
}
164+
49165
interface PkceCodes {
50166
verifier: string
51167
challenge: string
@@ -429,18 +545,19 @@ export async function CodexAuthPlugin(input: PluginInput): Promise<Hooks> {
429545
if (auth.type !== "oauth") return {}
430546

431547
// Filter models to only those the ChatGPT-subscription (Codex) tier
432-
// accepts. Delegates to ``shouldAllowOAuthModel`` (module-level,
433-
// above). See OAUTH_ALLOWED_MODELS + shouldAllowOAuthModel for the
434-
// criteria + how to add new gpt-5.N releases.
548+
// accepts. Delegates to ``disallowedOAuthModelKeys`` (module-level,
549+
// above), which matches on each model's upstream ``api.id`` so a
550+
// config alias of a supported model survives. See OAUTH_ALLOWED_MODELS
551+
// for the criteria + how to add new gpt-5.N releases.
435552
//
436553
// NOTE: this file is the ACTIVE plugin (wired via plugin/index.ts).
437554
// The sibling plugin/openai/codex.ts is an unwired in-progress
438555
// refactor that keeps its OWN ALLOWED_MODELS + parseFloat > 5.4
439556
// fallback — this filter does NOT share a source of truth with it.
440557
// Adopting shouldAllowOAuthModel there is followup on that refactor.
441558
// (Closes #1132 — GPT 5.6 missing from picker.)
442-
for (const modelId of Object.keys(provider.models)) {
443-
if (!shouldAllowOAuthModel(modelId)) delete provider.models[modelId]
559+
for (const modelId of disallowedOAuthModelKeys(provider.models)) {
560+
delete provider.models[modelId]
444561
}
445562

446563
// Zero out costs for Codex (included with ChatGPT subscription)

0 commit comments

Comments
 (0)