Skip to content

Commit f52b90d

Browse files
committed
test(auth): install the fake clock and fetch stub before startServer
startServer returns synchronously but arms an async pool-quota prime that outlives its return (src/server/index.ts:2054-2064). That prime calls getValidCodexToken, which can rotate the very credential these assertions read, and fetches a real host unless the stub is up. Both fixtures installed Date.now and globalThis.fetch AFTER startServer, leaving a window two dynamic import() resolutions wide where the prime ran against the real clock and real fetch. On a warm local module cache it resolved before the fixture finished; on a loaded CI runner it did not, and seenAuth[0] was already the rotated token. Measured rather than assumed: OPENCODEX_DEBUG_QUOTA=1 prints refreshed=1 on every run of both the fixed and unfixed trees, so the prime always fires. The fix does not suppress it -- it makes it run inside the fixture's controlled world. The thread-affinity test at :2131 had the identical shape and is fixed too.
1 parent afc6e78 commit f52b90d

2 files changed

Lines changed: 75 additions & 25 deletions

File tree

‎devlog/_plan/260901_merge_train_round3/060_wp7_websocket_refresh_flake.md‎

Lines changed: 53 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -108,22 +108,59 @@ src/codex/quota.ts:457
108108
The fixture calls `updateAccountQuota("pool-a", 10, 5)` at `:2242`, **before**
109109
`Date.now` is faked — so `updatedAt` is stamped with the **real** clock, 2026-09-01.
110110

111-
Now the two orderings diverge:
112-
113-
| when the prime's `Date.now()` runs | value | `Date.now() - updatedAt` | verdict |
114-
| --- | --- | --- | --- |
115-
| before `:2251` installs the fake clock | real, 2026-09-01 | ~0 ms | fresh — no fetch |
116-
| after `:2251` | fake, 2027-01-15 | ~136 days | **stale — fetch fires** |
117-
118-
So the failure needs the prime to land on the **late** side, not the early one. The stored
119-
credential is not near expiry under either clock; what varies is whether a quota fetch is
120-
triggered at all, and that fetch is what rotates the token before the first turn is served.
121-
122-
The earlier reading — "60 s of margin against `REFRESH_SKEW_MS`, the read lands on the
123-
wrong side of the boundary" — was wrong in both quantity and direction. The margin is months,
124-
and the fake clock does not shorten it; it inflates a *cache age* past a five-minute TTL.
125-
Recording that here because the wrong version is already quoted in comments on #3109 and
126-
#3112, and in `260901_release_train_2390/070_outcome.md`.
111+
Except that table was a prediction, and measuring it refuted the interesting half.
112+
113+
## Correction: the prime is ALWAYS stale, before and after the fix
114+
115+
```
116+
$ for i in 1..5: OPENCODEX_DEBUG_QUOTA=1 bun test ... -t "websocket passthrough refreshes pool auth"
117+
refreshed=1 before-fix run1 ... refreshed=1 before-fix run5
118+
refreshed=1 1 pass 0 fail run1 ... refreshed=1 1 pass 0 fail run5 (after fix)
119+
```
120+
121+
`refreshed=1` every single time, on both trees. So staleness never varied and the clock
122+
ordering is **not** the race. The predicted table is wrong.
123+
124+
What actually varies is what the prime's quota fetch *hits*:
125+
126+
```
127+
src/codex/auth-api.ts:1145-1158 (fetchFreshPoolAccountQuota)
128+
const { accessToken, chatgptAccountId, generation } = await getValidCodexToken(accountId);
129+
const resp = await fetch("https://chatgpt.com/backend-api/wham/usage", { ... });
130+
```
131+
132+
Two things happen there, and the fixture controls exactly one of them at `:2251`:
133+
134+
1. `getValidCodexToken` may **rotate the credential** — that is the token the assertion
135+
reads.
136+
2. `fetch` goes to a real host unless the stub is installed.
137+
138+
The stub was installed **after** `startServer`, so for the width of two dynamic
139+
`import()` resolutions the prime could reach the real `fetch` and the unpinned clock. Which
140+
of the two turns' credentials it left behind depended on whether it resolved before or after
141+
the fixture finished setting itself up — module-cache warmth and machine load, exactly the
142+
shape of a CI-only failure.
143+
144+
**So the fix is right for a reason one step over from the one first written down.** Moving
145+
the clock *and the fetch stub* above `startServer` does not stop the prime from running —
146+
`refreshed=1` still fires every run — it makes the prime run entirely inside the fixture's
147+
own controlled world, where its token refresh is served by the stub and its clock is the
148+
pinned one. The prime becomes deterministic instead of suppressed.
149+
150+
That distinction matters for anyone reading this later: if a future change makes the prime
151+
stop firing, this test is no longer covering what it thinks it covers.
152+
153+
Three explanations have now been written for this failure, and two of them were wrong:
154+
155+
| version | claim | verdict |
156+
| --- | --- | --- |
157+
| `260901_release_train_2390/070_outcome.md` | 60 s of margin against `REFRESH_SKEW_MS`; the read lands on the wrong side | wrong — the margin is months |
158+
| this doc, first pass | the fake clock inflates cache age past the TTL, so staleness varies | wrong — `refreshed=1` on every run of both trees |
159+
| this doc, measured | the prime always fetches; what varied was whether it hit the stubbed or the real `fetch`/clock | holds under measurement |
160+
161+
The first two were each plausible, each cited a real mechanism, and each would have justified
162+
the same fix. That is precisely why they were dangerous: a fix that works for the wrong reason
163+
teaches the wrong lesson to whoever touches it next.
127164

128165
## Why it will not reproduce locally
129166

‎tests/server-auth.test.ts‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2128,9 +2128,16 @@ describe("server local API auth", () => {
21282128
updateAccountQuota("pool-a", 10, 5);
21292129

21302130
const originalNow = Date.now;
2131+
// Pin the clock BEFORE startServer, not after. `startServer` returns synchronously but
2132+
// arms an async pool-quota prime (src/server/index.ts:2054-2064) that outlives its
2133+
// return, and that prime decides staleness with `Date.now() - quota.updatedAt >=
2134+
// POOL_CACHE_TTL` (src/codex/auth-api.ts:1334-1337). `updateAccountQuota` above stamped
2135+
// `updatedAt` with the REAL clock, so a prime that lands after a 2027 fake clock is
2136+
// installed sees months of cache age, fetches, and rotates the credential out from under
2137+
// the assertions. Installing the clock first closes the window entirely.
2138+
Date.now = () => now;
21312139
const server = startServer(0);
21322140
try {
2133-
Date.now = () => now;
21342141
for (const threadId of ["expired-http", "expired-compact", "expired-ws"]) {
21352142
const response = await fetch(new URL("/v1/responses", server.url), {
21362143
method: "POST",
@@ -2243,12 +2250,15 @@ describe("server local API auth", () => {
22432250

22442251
const originalNow = Date.now;
22452252
const originalFetch = globalThis.fetch;
2246-
const server = startServer(0);
2247-
const wsUrl = new URL("/v1/responses", server.url);
2248-
wsUrl.protocol = "ws:";
2249-
try {
2250-
Date.now = () => now;
2251-
globalThis.fetch = (async (input, init) => {
2253+
// Both the clock and the fetch stub go up before `startServer`. The async pool-quota
2254+
// prime it arms (src/server/index.ts:2054-2064) reads the clock AND fetches, so leaving
2255+
// either real for the width of two dynamic `import()` resolutions is what made this test
2256+
// fail on loaded CI runners while passing locally: the prime judged `pool-a` stale
2257+
// against a 2027 clock versus a `updatedAt` stamped in real time, then refreshed the
2258+
// credential before the first turn was served — so `seenAuth[0]` was already the new
2259+
// token. The failure diff was always the first element, never the second.
2260+
Date.now = () => now;
2261+
globalThis.fetch = (async (input, init) => {
22522262
const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url;
22532263
if (url === "https://auth.openai.com/oauth/token") {
22542264
return new Response(JSON.stringify({
@@ -2258,8 +2268,11 @@ describe("server local API auth", () => {
22582268
}), { status: 200 });
22592269
}
22602270
return originalFetch(input, init);
2261-
}) as typeof fetch;
2262-
2271+
}) as typeof fetch;
2272+
const server = startServer(0);
2273+
const wsUrl = new URL("/v1/responses", server.url);
2274+
wsUrl.protocol = "ws:";
2275+
try {
22632276
const ws = new WebSocket(wsUrl);
22642277
const waitForOpen = new Promise<void>((resolve, reject) => {
22652278
ws.addEventListener("open", () => resolve(), { once: true });

0 commit comments

Comments
 (0)