Skip to content

Commit 71f25bd

Browse files
anandgupta42claude
andcommitted
fix: correct the ChatGPT-subscription model allowlist against the live backend
The OAuth model filter was built from the models.dev catalog rather than from what the Codex endpoint actually serves, so it was wrong in both directions. Verified every gpt-5.x id against `POST /backend-api/codex/responses` on a ChatGPT Pro credential and rebuilt the allowlist from the results. Removed (offered in the picker, rejected by the backend with HTTP 400): - `gpt-5.2` and `gpt-5.6`, both explicitly allowlisted - `gpt-5.3-codex`, admitted by the `modelId.includes("codex")` auto-allow Added (served by the backend, previously hidden from the picker): - `gpt-5.6-sol`, `gpt-5.6-luna`, `gpt-5.6-terra` — the current flagship subscription models, excluded on an untested assumption that the sol/luna/terra variants were API-tier-only Dropped the `includes("codex")` substring auto-allow. It cannot express the real policy: the tier accepts `gpt-5.3-codex-spark` but rejects `gpt-5.3-codex`, and accepts `gpt-5.6-sol` but rejects `gpt-5.6`. Membership is now exact-match only. Transport was never involved. Plain HTTP with our own `originator: altimate` identity reaches every working model; no websocket, `openai-beta`, or `x-codex-beta-features` header changes any outcome. Rewrote `codex-allowlist.test.ts` around the verified truth table so both failure directions are guarded, plus a regression barrier against reintroducing a substring rule. Closes #1178 Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ
1 parent 23e5903 commit 71f25bd

2 files changed

Lines changed: 111 additions & 99 deletions

File tree

‎packages/opencode/src/plugin/codex.ts‎

Lines changed: 37 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -16,33 +16,52 @@ const CODEX_API_ENDPOINT = "https://chatgpt.com/backend-api/codex/responses"
1616
const OAUTH_PORT = 1455
1717
const OAUTH_POLLING_SAFETY_MARGIN_MS = 3000
1818

19-
/** Non-codex ChatGPT-subscription (OAuth) allowlist. Any modelId
20-
* containing "codex" is auto-allowed by ``shouldAllowOAuthModel`` below,
21-
* so this set only enumerates the plain non-codex main/mini variants
22-
* OpenAI exposes on Codex-tier accounts. Bump whenever a new gpt-5.N
23-
* is generally available on the subscription. Exported for unit-test
24-
* coverage — see test/plugin/codex-allowlist.test.ts. */
19+
/** Exact set of model ids the ChatGPT-subscription (Codex) tier accepts.
20+
*
21+
* Every entry was verified against the live backend
22+
* (POST https://chatgpt.com/backend-api/codex/responses) on a ChatGPT Pro
23+
* credential: entries here returned HTTP 200, and every other gpt-5.x id in
24+
* the models.dev catalog returned
25+
* ``400 {"detail":"The '<id>' model is not supported when using Codex with a
26+
* ChatGPT account."}``.
27+
*
28+
* Confirmed rejected, so deliberately absent: gpt-5, gpt-5.1, gpt-5.2,
29+
* gpt-5.2-pro, gpt-5.3-chat-latest, gpt-5.3-codex, gpt-5.4-nano, gpt-5.4-pro,
30+
* gpt-5.5-pro, gpt-5.6.
31+
*
32+
* There is no derivable rule here — the tier accepts ``gpt-5.3-codex-spark``
33+
* but rejects ``gpt-5.3-codex``, and accepts the gpt-5.6 sol/luna/terra
34+
* variants but rejects plain ``gpt-5.6``. So this is an exact-match list by
35+
* necessity, not by preference. Only add an id after confirming a 200 from the
36+
* endpoint above on a subscription credential; guessing puts a model in the
37+
* picker that then fails at request time.
38+
*
39+
* Exported for unit-test coverage — see test/plugin/codex-allowlist.test.ts. */
2540
export const OAUTH_ALLOWED_MODELS = new Set([
26-
"gpt-5.2",
41+
"gpt-5.3-codex-spark",
2742
"gpt-5.4",
2843
"gpt-5.4-mini",
2944
"gpt-5.5",
30-
"gpt-5.6",
45+
"gpt-5.6-luna",
46+
"gpt-5.6-sol",
47+
"gpt-5.6-terra",
3148
])
3249

3350
/** OAuth (ChatGPT-subscription) model-filter policy for the ACTIVE plugin
34-
* (this file — wired via plugin/index.ts). A model is kept if either
35-
* (a) its id contains ``"codex"`` (all codex variants ship on the
36-
* subscription), or (b) its id is an exact member of
37-
* ``OAUTH_ALLOWED_MODELS`` (the curated non-codex releases).
51+
* (this file — wired via plugin/index.ts). Exact membership in
52+
* ``OAUTH_ALLOWED_MODELS`` — no substring heuristics.
53+
*
54+
* This previously auto-allowed any id containing ``"codex"``, which admitted
55+
* ``gpt-5.3-codex``. The backend rejects that id, so it reached the picker and
56+
* then failed at request time with an HTTP 400. Substring matching cannot
57+
* express the real policy (``gpt-5.3-codex-spark`` is accepted while
58+
* ``gpt-5.3-codex`` is not), so the heuristic is gone.
3859
*
39-
* The sibling file plugin/openai/codex.ts (an in-progress refactor,
40-
* currently NOT wired) has its own separate filter with a
41-
* ``parseFloat(match[1]) > 5.4`` fallback. Adopting this helper is
42-
* followup work on that refactor — do NOT assume the two files share
43-
* this policy today. */
60+
* The sibling file plugin/openai/codex.ts (an in-progress refactor, currently
61+
* NOT wired) has its own separate filter with a ``parseFloat(match[1]) > 5.4``
62+
* fallback. Adopting this helper is followup work on that refactor — do NOT
63+
* assume the two files share this policy today. */
4464
export function shouldAllowOAuthModel(modelId: string): boolean {
45-
if (modelId.includes("codex")) return true
4665
return OAUTH_ALLOWED_MODELS.has(modelId)
4766
}
4867

‎packages/opencode/test/plugin/codex-allowlist.test.ts‎

Lines changed: 74 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,24 @@
22
// in packages/opencode/src/plugin/codex.ts — the ACTIVE plugin wired
33
// via plugin/index.ts.
44
//
5-
// Filed as issue #1132: GPT 5.6 was released but the allowlist stopped
6-
// at 5.4, so users on ChatGPT Pro/Plus (Codex tier) couldn't pick it in
7-
// the model picker even though the underlying models.dev catalog had it.
5+
// The allowlist is a verified truth table, not a guess. Every id asserted
6+
// below was probed against the live backend
7+
// (POST https://chatgpt.com/backend-api/codex/responses) on a ChatGPT Pro
8+
// credential, using our own `originator: altimate` client identity over plain
9+
// HTTP. Accepted ids returned HTTP 200; rejected ids returned
10+
// 400 {"detail":"The '<id>' model is not supported when using Codex with a
11+
// ChatGPT account."}
12+
//
13+
// Two directions of breakage this file guards:
14+
// * false positives — an id offered in the picker that 400s at request time
15+
// (previously gpt-5.2, gpt-5.6, gpt-5.3-codex)
16+
// * false negatives — a working subscription model hidden from the picker
17+
// (previously the gpt-5.6 sol/luna/terra variants, i.e. the current
18+
// flagship models)
19+
//
20+
// Supersedes the narrower framing of issue #1132, which added plain gpt-5.6 to
21+
// the allowlist on catalog presence alone. The backend rejects that id; only
22+
// its sol/luna/terra variants are actually served on the subscription.
823
//
924
// Sibling file plugin/openai/codex.ts is an in-progress refactor of the
1025
// same plugin, currently NOT wired via plugin/index.ts, and NOT covered
@@ -17,112 +32,90 @@
1732
import { describe, expect, test } from "bun:test"
1833
import { OAUTH_ALLOWED_MODELS, shouldAllowOAuthModel } from "../../src/plugin/codex"
1934

20-
describe("OAUTH_ALLOWED_MODELS — subscription model picker regression barrier", () => {
21-
test("issue #1132: gpt-5.6 is present", () => {
22-
// If this fails again, someone dropped gpt-5.6 from the non-codex
23-
// allowlist without moving forward to a newer generation — rejecting
24-
// a shipped OpenAI model users have subscription access to.
25-
expect(OAUTH_ALLOWED_MODELS.has("gpt-5.6")).toBe(true)
26-
})
35+
/** Verified HTTP 200 on a ChatGPT Pro subscription credential. */
36+
const VERIFIED_ACCEPTED = [
37+
"gpt-5.3-codex-spark",
38+
"gpt-5.4",
39+
"gpt-5.4-mini",
40+
"gpt-5.5",
41+
"gpt-5.6-luna",
42+
"gpt-5.6-sol",
43+
"gpt-5.6-terra",
44+
]
2745

28-
test("gpt-5.5 is present (added alongside 5.6 for parity)", () => {
29-
expect(OAUTH_ALLOWED_MODELS.has("gpt-5.5")).toBe(true)
30-
})
46+
/** Verified HTTP 400 "not supported when using Codex with a ChatGPT account". */
47+
const VERIFIED_REJECTED = [
48+
"gpt-5",
49+
"gpt-5.1",
50+
"gpt-5.2",
51+
"gpt-5.2-pro",
52+
"gpt-5.3-chat-latest",
53+
"gpt-5.3-codex",
54+
"gpt-5.4-nano",
55+
"gpt-5.4-pro",
56+
"gpt-5.5-pro",
57+
"gpt-5.6",
58+
]
3159

32-
test("prior non-codex generations stay allowlisted (no accidental removal)", () => {
33-
for (const id of ["gpt-5.2", "gpt-5.4", "gpt-5.4-mini"]) {
60+
describe("OAUTH_ALLOWED_MODELS — verified subscription truth table", () => {
61+
test("every id verified as accepted is allowlisted", () => {
62+
// False negatives hide working models from the picker. The sol/luna/terra
63+
// variants are the current flagship subscription models — a previous
64+
// revision excluded them on the untested assumption that they were
65+
// API-tier-only, which cost users access to models they already pay for.
66+
for (const id of VERIFIED_ACCEPTED) {
3467
expect(OAUTH_ALLOWED_MODELS.has(id)).toBe(true)
3568
}
3669
})
3770

38-
test("codex-tagged variants are NOT in the non-codex set (they're auto-allowed instead)", () => {
39-
// The non-codex set is deliberately minimal — every codex-tagged id
40-
// is auto-allowed by shouldAllowOAuthModel's `includes("codex")` check
41-
// below, so listing them here would be redundant + a maintenance trap.
42-
for (const id of ["gpt-5.1-codex", "gpt-5.2-codex", "gpt-5.3-codex"]) {
43-
expect(OAUTH_ALLOWED_MODELS.has(id)).toBe(false)
44-
}
45-
})
46-
47-
test("allowlist does NOT include API-tier-only variants (defensive)", () => {
48-
// Pro / luna / sol / terra variants ship on models.dev but aren't
49-
// confirmed available on the ChatGPT-subscription (Codex) tier —
50-
// showing them in the picker would surface a request-time failure.
51-
// If OpenAI extends subscription coverage to them, add them here
52-
// deliberately (with a link to the announcement).
53-
for (const id of [
54-
"gpt-5.4-pro",
55-
"gpt-5.5-pro",
56-
"gpt-5.6-luna",
57-
"gpt-5.6-sol",
58-
"gpt-5.6-terra",
59-
]) {
71+
test("every id verified as rejected stays out of the allowlist", () => {
72+
// False positives are worse than a missing model: the id shows up in the
73+
// picker, the user selects it, and the request dies with an opaque 400.
74+
for (const id of VERIFIED_REJECTED) {
6075
expect(OAUTH_ALLOWED_MODELS.has(id)).toBe(false)
6176
}
6277
})
6378

64-
test("allowlist size never regresses below current baseline", () => {
65-
// Trip-wire: if someone truncates the allowlist by mistake (or in
66-
// a bad rebase), the count drops and this test catches it before
67-
// shipping. Bump when a real new addition lands.
68-
expect(OAUTH_ALLOWED_MODELS.size).toBeGreaterThanOrEqual(5)
79+
test("the allowlist contains nothing beyond the verified-accepted set", () => {
80+
// Trip-wire against speculative additions. To add an id here, probe it
81+
// against the live endpoint first and land it in VERIFIED_ACCEPTED too.
82+
expect([...OAUTH_ALLOWED_MODELS].sort()).toEqual([...VERIFIED_ACCEPTED].sort())
6983
})
7084
})
7185

7286
describe("shouldAllowOAuthModel — behavior of the filter itself", () => {
7387
// Behavior-level coverage: even if a refactor stops passing
7488
// OAUTH_ALLOWED_MODELS through, the filter function is what the
7589
// loader actually calls, so this catches breakage the constant-only
76-
// tests above would miss. (cubic P3 catch.)
90+
// tests above would miss.
7791

78-
test("allowlist members pass (spot-check each generation)", () => {
79-
for (const id of ["gpt-5.2", "gpt-5.4", "gpt-5.4-mini", "gpt-5.5", "gpt-5.6"]) {
92+
test("verified-accepted ids pass", () => {
93+
for (const id of VERIFIED_ACCEPTED) {
8094
expect(shouldAllowOAuthModel(id)).toBe(true)
8195
}
8296
})
8397

84-
test("codex-tagged ids pass regardless of exact allowlist membership", () => {
85-
// Any id containing "codex" auto-passes — covers gpt-5.1-codex,
86-
// gpt-5.3-codex-spark, plus any future codex variant OpenAI ships.
87-
for (const id of [
88-
"gpt-5.1-codex",
89-
"gpt-5.1-codex-max",
90-
"gpt-5.1-codex-mini",
91-
"gpt-5.2-codex",
92-
"gpt-5.3-codex",
93-
"gpt-5.3-codex-spark",
94-
"gpt-5.3-codex-xhigh",
95-
"codex-hypothetical-future-name",
96-
]) {
97-
expect(shouldAllowOAuthModel(id)).toBe(true)
98+
test("verified-rejected ids are filtered out", () => {
99+
for (const id of VERIFIED_REJECTED) {
100+
expect(shouldAllowOAuthModel(id)).toBe(false)
98101
}
99102
})
100103

101-
test("API-tier-only variants are rejected (the whole point of the filter)", () => {
102-
// These are the models the previous parseFloat > 5.4 fallback in
103-
// plugin/openai/codex.ts was incorrectly admitting; the shared
104-
// filter must reject them so the picker stays honest about what
105-
// the subscription actually accepts.
106-
for (const id of [
107-
"gpt-5.4-pro",
108-
"gpt-5.4-nano",
109-
"gpt-5.5-pro",
110-
"gpt-5.6-luna",
111-
"gpt-5.6-sol",
112-
"gpt-5.6-terra",
113-
]) {
114-
expect(shouldAllowOAuthModel(id)).toBe(false)
115-
}
104+
test("a 'codex' substring does not grant access on its own", () => {
105+
// Regression barrier for the removed `modelId.includes("codex")`
106+
// auto-allow. The backend accepts gpt-5.3-codex-spark but rejects
107+
// gpt-5.3-codex, so no substring rule can express the real policy —
108+
// reintroducing one puts broken ids back in the picker.
109+
expect(shouldAllowOAuthModel("gpt-5.3-codex")).toBe(false)
110+
expect(shouldAllowOAuthModel("gpt-5.1-codex")).toBe(false)
111+
expect(shouldAllowOAuthModel("gpt-5.1-codex-max")).toBe(false)
112+
expect(shouldAllowOAuthModel("codex-hypothetical-future-name")).toBe(false)
113+
// ...while the one codex id that IS accepted still passes, by exact match.
114+
expect(shouldAllowOAuthModel("gpt-5.3-codex-spark")).toBe(true)
116115
})
117116

118117
test("completely unrelated ids are rejected", () => {
119-
for (const id of [
120-
"claude-3.5-sonnet",
121-
"gemini-2.5-pro",
122-
"gpt-4o",
123-
"gpt-4-turbo",
124-
"",
125-
]) {
118+
for (const id of ["claude-3.5-sonnet", "gemini-2.5-pro", "gpt-4o", "gpt-4-turbo", ""]) {
126119
expect(shouldAllowOAuthModel(id)).toBe(false)
127120
}
128121
})

0 commit comments

Comments
 (0)