Skip to content

Commit 35d28a0

Browse files
committed
fix: classify codex upstream outcomes
1 parent 820fd8c commit 35d28a0

5 files changed

Lines changed: 370 additions & 7 deletions

File tree

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# 50B - Phase 50B Plan: Codex Upstream Outcome Taxonomy Foundation
2+
3+
Date: 2026-06-24
4+
5+
Status: implemented, focused-verified.
6+
7+
## Objective
8+
9+
Continue Patch 5 from `/Users/jun/Developer/new/700_projects/opencodex/devlog/280_codex-multi-auth-security-patch-plan/00_patch_plan.md` by replacing the current "every non-2xx is an account failure" behavior with a small explicit outcome taxonomy.
10+
11+
This phase is intentionally narrow:
12+
13+
- caller/model 4xx responses do not penalize the selected Codex account;
14+
- credential failures immediately quarantine the selected Codex account;
15+
- 5xx and provider-connect failures count toward transient failover;
16+
- passthrough connect failures are recorded before returning the local 502, with a server-level integration test;
17+
- 429 and terminal SSE/WebSocket application failures remain documented follow-up slices.
18+
19+
## External Basis
20+
21+
RFC 9110 separates HTTP status classes by semantics: 4xx is client error, 5xx is server error, 401 is authentication-specific, and Retry-After can accompany temporary unavailability or rate limiting. That means account health must not treat ordinary caller/model 400/404/422 responses as equivalent to provider outage or credential failure.
22+
23+
References checked:
24+
25+
- RFC 9110 HTTP Semantics: https://datatracker.ietf.org/doc/html/rfc9110
26+
27+
## Acceptance Criteria
28+
29+
- `recordCodexUpstreamOutcome(config, accountId, 400 | 404 | 422)` does not increment upstream health and does not trigger account failover.
30+
- `recordCodexUpstreamOutcome(config, accountId, 401)` marks the account as needing reauth and immediately makes future new threads pick another usable pool account when one exists.
31+
- `recordCodexUpstreamOutcome(config, accountId, 403)` is treated as credential-specific for now and also quarantines the account.
32+
- `recordCodexUpstreamOutcome(config, accountId, 500 | 502 | 503 | 504)` increments transient health and still honors `upstreamFailoverThreshold`.
33+
- `recordCodexUpstreamOutcome(config, accountId, "connect_error")` or equivalent explicit connect outcome increments transient health.
34+
- Passthrough upstream fetch failures in `src/server.ts` record a connect failure for the selected pool account before returning the local `502`.
35+
- The server catch behavior is verified through `startServer(0)` and a loopback `POST /v1/responses` request against an unreachable passthrough base URL.
36+
- Existing 2xx reset behavior remains unchanged.
37+
- The public request log still uses safe provider labels only; no local account IDs or emails are introduced.
38+
39+
## File Plan
40+
41+
### MODIFY `/Users/jun/Developer/new/700_projects/opencodex/src/codex-routing.ts`
42+
43+
Add a small exported taxonomy:
44+
45+
```ts
46+
export type CodexUpstreamOutcome =
47+
| number
48+
| "connect_error"
49+
| "timeout";
50+
51+
export type CodexUpstreamOutcomeClass =
52+
| "success"
53+
| "credential"
54+
| "quota"
55+
| "transient"
56+
| "caller"
57+
| "unknown";
58+
59+
export function classifyCodexUpstreamOutcome(outcome: CodexUpstreamOutcome): CodexUpstreamOutcomeClass
60+
```
61+
62+
Classification rules:
63+
64+
- status `200..299` => `success`;
65+
- `401` and `403` => `credential`;
66+
- `429` => `quota` but no cooldown behavior yet in this phase;
67+
- `400..499` => `caller`;
68+
- `500..599`, `"connect_error"`, and `"timeout"` => `transient`;
69+
- everything else => `unknown`.
70+
71+
Change `recordCodexUpstreamOutcome()` to:
72+
73+
- reset health on `success`;
74+
- ignore `caller` outcomes entirely;
75+
- mark reauth and clear thread affinity on `credential`;
76+
- increment upstream health for `transient`, `quota`, and `unknown`;
77+
- for credential outcomes, do not rely on `applyFailureFailover()` because it is threshold-based; instead make the account unusable by reauth marking and clear its affinities, so the next `resolveCodexAccountForThread()` call selects another usable pool account;
78+
- for transient/quota/unknown outcomes, apply normal threshold-based failover after increment if the selected account is active.
79+
80+
The `upstreamHealth.lastFailureStatus` field can remain numeric for statuses and use `0` for connect/timeout sentinels to avoid widening unrelated API shapes in this slice.
81+
82+
### MODIFY `/Users/jun/Developer/new/700_projects/opencodex/src/server.ts`
83+
84+
In passthrough `fetchWithHeaderTimeout()` catch:
85+
86+
- before returning `formatErrorResponse(502, ...)`, call `recordCodexUpstreamOutcome()` when `authCtx.kind === "pool"`;
87+
- use `"timeout"` when the thrown error name is `TimeoutError`;
88+
- use `"connect_error"` otherwise.
89+
90+
Keep the response body and status unchanged.
91+
92+
### MODIFY `/Users/jun/Developer/new/700_projects/opencodex/tests/codex-routing.test.ts`
93+
94+
Import `clearAccountNeedsReauth` from `/Users/jun/Developer/new/700_projects/opencodex/src/codex-auth-api.ts` and clear reauth state for test pool ids in `beforeEach()` and `afterEach()` because credential-outcome tests intentionally mark accounts as reauth-needed.
95+
96+
Update the old test named `three consecutive non-200 responses fail over future new threads` to `three consecutive transient failures fail over future new threads`.
97+
98+
Add tests:
99+
100+
- caller/model 4xx responses are ignored for health/failover;
101+
- 401 quarantines an account immediately and marks reauth;
102+
- 403 quarantines the account under the current conservative credential policy;
103+
- connect failures contribute to failover;
104+
- classifier returns the expected labels for representative status codes and connect sentinels.
105+
106+
### MODIFY `/Users/jun/Developer/new/700_projects/opencodex/tests/server-auth.test.ts`
107+
108+
Add a server integration test:
109+
110+
- create a temporary `OPENCODEX_HOME`;
111+
- save a config with default provider `chatgpt` using the `openai-responses` passthrough adapter and an unreachable loopback port;
112+
- save pool credentials and set `activeCodexAccountId` to that pool account;
113+
- start the proxy with `startServer(0)`;
114+
- `POST /v1/responses` to `server.url` with loopback origin and a small valid Responses body;
115+
- assert the response is `502`;
116+
- assert `getCodexUpstreamHealth(poolId)` shows one recorded connect/timeout failure;
117+
- stop the server and restore environment state.
118+
119+
This avoids exporting `handleResponses()` only for tests.
120+
121+
### OPTIONAL MODIFY `/Users/jun/Developer/new/700_projects/opencodex/tests/codex-auth-context.test.ts`
122+
123+
No planned change unless focused routing tests expose reauth-state cleanup needs.
124+
125+
## Verification Plan
126+
127+
Focused:
128+
129+
```bash
130+
bun test tests/codex-routing.test.ts tests/server-auth.test.ts
131+
```
132+
133+
Full:
134+
135+
```bash
136+
bun run typecheck
137+
bun test tests
138+
cd gui && bun run build
139+
git diff --check
140+
git status --short
141+
```
142+
143+
Independent verification:
144+
145+
- dispatch a read-only Backend/Security verifier to confirm caller 4xx no longer burns account health, credential failures fail closed/quarantine, and connect failures now count toward failover without exposing account identifiers.
146+
147+
## Implementation Evidence
148+
149+
Changed files:
150+
151+
- `/Users/jun/Developer/new/700_projects/opencodex/src/codex-routing.ts`
152+
- `/Users/jun/Developer/new/700_projects/opencodex/src/server.ts`
153+
- `/Users/jun/Developer/new/700_projects/opencodex/tests/codex-routing.test.ts`
154+
- `/Users/jun/Developer/new/700_projects/opencodex/tests/server-auth.test.ts`
155+
156+
Implemented:
157+
158+
- Added `CodexUpstreamOutcome`, `CodexUpstreamOutcomeClass`, and `classifyCodexUpstreamOutcome()`.
159+
- `recordCodexUpstreamOutcome()` now ignores caller/model 4xx outcomes, quarantines 401/403 credential outcomes with reauth marking plus thread-affinity purge, and records connect/timeout sentinels as transient failures.
160+
- Passthrough connect/timeout failures in `src/server.ts` now record selected pool account health before returning local `502`.
161+
- Routing tests now clear reauth runtime state and cover classifier labels, caller 4xx no-penalty behavior, 401/403 quarantine, and connect-failure failover.
162+
- Server integration test now proves the passthrough connect catch records pool account health through `startServer(0)` and `POST /v1/responses`.
163+
164+
Focused verification:
165+
166+
```text
167+
bun test tests/codex-routing.test.ts tests/server-auth.test.ts
168+
27 pass
169+
0 fail
170+
70 expect() calls
171+
```
172+
173+
## Out Of Scope For This Slice
174+
175+
- 429 cooldown window using `Retry-After` or quota reset metadata.
176+
- Time-bounded failure streak expiry using `lastFailureAt`.
177+
- Recording terminal SSE or WebSocket `response.failed` / `response.incomplete` after stream completion.
178+
- Selected-pool sidecar outcome recording for web-search and vision sidecars; this needs a separate hook across `/Users/jun/Developer/new/700_projects/opencodex/src/web-search/executor.ts` and `/Users/jun/Developer/new/700_projects/opencodex/src/vision/describe.ts` and will be handled as a follow-up Patch 5 slice.
179+
- WebSocket auth-context consistency beyond already-merged fail-closed handling.
180+
181+
## Plan Audit Fixes
182+
183+
- Added reauth-state cleanup to the routing test plan because 401/403 tests mutate module-level runtime state.
184+
- Added a `startServer(0)` integration test plan for the passthrough connect-failure catch so server wiring is verified, not only the routing helper.
185+
- Clarified that credential quarantine must not rely on threshold-based `applyFailureFailover()`.
186+
- Explicitly scoped sidecar account-outcome recording out of Phase 50B and into a follow-up Patch 5 slice.

‎src/codex-routing.ts‎

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ import type { OcxConfig } from "./types";
77
const threadAccountMap = new Map<string, string>();
88
const upstreamHealth = new Map<string, { consecutiveFailures: number; lastFailureStatus?: number; lastFailureAt?: number }>();
99

10+
export type CodexUpstreamOutcome = number | "connect_error" | "timeout";
11+
export type CodexUpstreamOutcomeClass = "success" | "credential" | "quota" | "transient" | "caller" | "unknown";
12+
1013
function hasConfiguredPoolAccount(config: OcxConfig, accountId: string): boolean {
1114
return (config.codexAccounts ?? []).some(account => !account.isMain && account.id === accountId);
1215
}
@@ -46,6 +49,17 @@ export function computeCodexUsageScore(quota: {
4649
return values.length > 0 ? Math.max(...values) : CODEX_UNKNOWN_USAGE_SCORE;
4750
}
4851

52+
export function classifyCodexUpstreamOutcome(outcome: CodexUpstreamOutcome): CodexUpstreamOutcomeClass {
53+
if (outcome === "connect_error" || outcome === "timeout") return "transient";
54+
if (!Number.isFinite(outcome)) return "unknown";
55+
if (outcome >= 200 && outcome < 300) return "success";
56+
if (outcome === 401 || outcome === 403) return "credential";
57+
if (outcome === 429) return "quota";
58+
if (outcome >= 400 && outcome < 500) return "caller";
59+
if (outcome >= 500 && outcome < 600) return "transient";
60+
return "unknown";
61+
}
62+
4963
function getEligiblePoolAccounts(config: OcxConfig, excludeId?: string): string[] {
5064
return (config.codexAccounts ?? [])
5165
.filter(account => !account.isMain && account.id !== excludeId && !isAccountNeedsReauth(account.id))
@@ -142,19 +156,33 @@ export function resolveCodexAccountForThread(
142156
return active;
143157
}
144158

145-
export function recordCodexUpstreamOutcome(config: OcxConfig, accountId: string | null, status: number): void {
159+
export function recordCodexUpstreamOutcome(config: OcxConfig, accountId: string | null, outcome: CodexUpstreamOutcome): void {
146160
if (!accountId) return;
147-
if (status >= 200 && status < 300) {
161+
const outcomeClass = classifyCodexUpstreamOutcome(outcome);
162+
if (outcomeClass === "success") {
148163
upstreamHealth.delete(accountId);
149164
return;
150165
}
166+
if (outcomeClass === "caller") return;
167+
168+
const lastFailureStatus = typeof outcome === "number" ? outcome : 0;
169+
if (outcomeClass === "credential") {
170+
upstreamHealth.set(accountId, {
171+
consecutiveFailures: 1,
172+
lastFailureStatus,
173+
lastFailureAt: Date.now(),
174+
});
175+
markAccountNeedsReauth(accountId);
176+
clearThreadAccountMapForAccount(accountId);
177+
return;
178+
}
179+
151180
const current = upstreamHealth.get(accountId);
152181
upstreamHealth.set(accountId, {
153182
consecutiveFailures: (current?.consecutiveFailures ?? 0) + 1,
154-
lastFailureStatus: status,
183+
lastFailureStatus,
155184
lastFailureAt: Date.now(),
156185
});
157-
if (status === 401) markAccountNeedsReauth(accountId);
158186
if (config.activeCodexAccountId === accountId) applyFailureFailover(config, accountId);
159187
}
160188

‎src/server.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,9 @@ async function handleResponses(
238238
}, upstream.signal, connectMs);
239239
} catch (err) {
240240
upstream.abort();
241-
const msg = err instanceof Error && err.name === "TimeoutError"
241+
const outcome = err instanceof Error && err.name === "TimeoutError" ? "timeout" : "connect_error";
242+
if (authCtx.kind === "pool") recordCodexUpstreamOutcome(config, authCtx.accountId, outcome);
243+
const msg = outcome === "timeout"
242244
? `Provider connect timeout after ${connectMs}ms`
243245
: `Provider unreachable: ${err instanceof Error ? err.message : String(err)}`;
244246
return formatErrorResponse(502, "upstream_error", msg);

‎tests/codex-routing.test.ts‎

Lines changed: 76 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import { describe, expect, test, beforeEach, afterEach } from "bun:test";
22
import { existsSync, mkdirSync, rmSync } from "node:fs";
33
import { join } from "node:path";
44
import {
5+
classifyCodexUpstreamOutcome,
56
clearCodexUpstreamHealth,
67
clearCodexUpstreamHealthForAccount,
78
clearThreadAccountMap,
@@ -13,7 +14,14 @@ import {
1314
resolveCodexAccountForThread,
1415
} from "../src/codex-routing";
1516
import { removeCodexAccountCredential, saveCodexAccountCredential } from "../src/codex-account-store";
16-
import { clearAccountQuota, handleCodexAuthAPI, parseUsageQuota, updateAccountQuota } from "../src/codex-auth-api";
17+
import {
18+
clearAccountNeedsReauth,
19+
clearAccountQuota,
20+
handleCodexAuthAPI,
21+
isAccountNeedsReauth,
22+
parseUsageQuota,
23+
updateAccountQuota,
24+
} from "../src/codex-auth-api";
1725
import { CODEX_UNKNOWN_USAGE_SCORE } from "../src/codex-quota";
1826
import type { OcxConfig } from "../src/types";
1927

@@ -52,6 +60,9 @@ describe("codex routing", () => {
5260
clearThreadAccountMap();
5361
clearCodexUpstreamHealth();
5462
clearAccountQuota();
63+
clearAccountNeedsReauth("a");
64+
clearAccountNeedsReauth("b");
65+
clearAccountNeedsReauth("c");
5566
saveTestCredential("a");
5667
saveTestCredential("b");
5768
});
@@ -60,6 +71,9 @@ describe("codex routing", () => {
6071
clearAccountQuota();
6172
clearCodexUpstreamHealth();
6273
clearThreadAccountMap();
74+
clearAccountNeedsReauth("a");
75+
clearAccountNeedsReauth("b");
76+
clearAccountNeedsReauth("c");
6377
if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME;
6478
else process.env.OPENCODEX_HOME = previousOpencodexHome;
6579
if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true });
@@ -102,7 +116,19 @@ describe("codex routing", () => {
102116
expect(pickLowestUsageCodexAccount(config)).toBe("b");
103117
});
104118

105-
test("three consecutive non-200 responses fail over future new threads", () => {
119+
test("upstream outcome classifier separates caller, credential, and transient failures", () => {
120+
expect(classifyCodexUpstreamOutcome(200)).toBe("success");
121+
expect(classifyCodexUpstreamOutcome(401)).toBe("credential");
122+
expect(classifyCodexUpstreamOutcome(403)).toBe("credential");
123+
expect(classifyCodexUpstreamOutcome(429)).toBe("quota");
124+
expect(classifyCodexUpstreamOutcome(422)).toBe("caller");
125+
expect(classifyCodexUpstreamOutcome(503)).toBe("transient");
126+
expect(classifyCodexUpstreamOutcome("connect_error")).toBe("transient");
127+
expect(classifyCodexUpstreamOutcome("timeout")).toBe("transient");
128+
expect(classifyCodexUpstreamOutcome(102)).toBe("unknown");
129+
});
130+
131+
test("three consecutive transient failures fail over future new threads", () => {
106132
const config = makeConfig();
107133
updateAccountQuota("a", 10, 5);
108134
updateAccountQuota("b", 20, 10);
@@ -114,6 +140,54 @@ describe("codex routing", () => {
114140
expect(resolveCodexAccountForThread("next", config)).toBe("b");
115141
});
116142

143+
test("caller and model 4xx responses do not penalize account health", () => {
144+
const config = makeConfig();
145+
updateAccountQuota("a", 10, 5);
146+
updateAccountQuota("b", 20, 10);
147+
recordCodexUpstreamOutcome(config, "a", 400);
148+
recordCodexUpstreamOutcome(config, "a", 404);
149+
recordCodexUpstreamOutcome(config, "a", 422);
150+
expect(getCodexUpstreamHealth("a")).toBeNull();
151+
expect(resolveCodexAccountForThread("next", config)).toBe("a");
152+
});
153+
154+
test("401 credential outcome quarantines the account for future threads", () => {
155+
const config = makeConfig();
156+
updateAccountQuota("a", 10, 5);
157+
updateAccountQuota("b", 20, 10);
158+
expect(resolveCodexAccountForThread("credential-existing", config)).toBe("a");
159+
160+
recordCodexUpstreamOutcome(config, "a", 401);
161+
162+
expect(isAccountNeedsReauth("a")).toBe(true);
163+
expect(getCodexUpstreamHealth("a")).toMatchObject({ consecutiveFailures: 1, lastFailureStatus: 401 });
164+
expect(resolveCodexAccountForThread("credential-existing", config)).toBe("b");
165+
expect(resolveCodexAccountForThread("credential-next", config)).toBe("b");
166+
});
167+
168+
test("403 credential outcome quarantines the account under the conservative policy", () => {
169+
const config = makeConfig();
170+
updateAccountQuota("a", 10, 5);
171+
updateAccountQuota("b", 20, 10);
172+
173+
recordCodexUpstreamOutcome(config, "a", 403);
174+
175+
expect(isAccountNeedsReauth("a")).toBe(true);
176+
expect(getCodexUpstreamHealth("a")).toMatchObject({ consecutiveFailures: 1, lastFailureStatus: 403 });
177+
expect(resolveCodexAccountForThread("credential-403-next", config)).toBe("b");
178+
});
179+
180+
test("connect failures contribute to transient failover", () => {
181+
const config = makeConfig();
182+
updateAccountQuota("a", 10, 5);
183+
updateAccountQuota("b", 20, 10);
184+
recordCodexUpstreamOutcome(config, "a", "connect_error");
185+
recordCodexUpstreamOutcome(config, "a", "timeout");
186+
recordCodexUpstreamOutcome(config, "a", "connect_error");
187+
expect(getCodexUpstreamHealth("a")).toMatchObject({ consecutiveFailures: 3, lastFailureStatus: 0 });
188+
expect(resolveCodexAccountForThread("connect-next", config)).toBe("b");
189+
});
190+
117191
test("2xx responses reset the failure streak", () => {
118192
const config = makeConfig();
119193
recordCodexUpstreamOutcome(config, "a", 503);

0 commit comments

Comments
 (0)