Skip to content

Commit 19b0157

Browse files
lidge-junjun
andauthored
fix(windows): server.stop drains the config-dir ACL flight; retrying test teardown (#3257)
* devlog(windows): CI failure inventory and repair roadmap (000-040) * devlog(windows): audit fixes — test filename, win32 platform seam in shutdown test, exact line cites * fix(windows): server.stop drains the config-dir ACL flight it started hardenConfigDir() spawns icacls.exe as a fire-and-forget flight (e5d5886). The child holds the config directory open until it exits, and Windows locks are mandatory, so removing the directory right after a clean server.stop() returned EPERM/EBUSY — which is what every Windows CI shard has hit since 2026-08-30 in the account-store, auth-api and live-server fixtures. flushConfigDirHardening(dir) is now a production function scoped to one directory; startServer captures its config dir before loadConfig and the composite stop() awaits the flight after the listeners close. Regression test drives stop() with a held flight and proves it stays pending until release (driven red before the fix). * test(account-store): per-test scratch home, both icacls runners stubbed, retrying teardown The fixed repo-local .tmp-codex-accounts-test meant one EPERM teardown poisoned every later case (49/49 errors in run 33590540220 were hooks). The sync-only icacls stub also no longer covered hardenConfigDir(), which uses the async runner since e5d5886. * test(windows): retrying scratch-home teardown across the live-server suites; per-test homes for auth-api and oauth-status Fourteen suites removed their scratch home with a bare rmSync right after server.stop(). Even with stop() now draining the ACL flight, antivirus and indexer handles on a fresh temp dir are real on the hosted image, so every teardown goes through removeTreeWithRetry (EPERM/EBUSY/ENOTEMPTY, bounded). codex-auth-api and oauth-status-privacy also move off fixed repo-local dirs and stub both icacls runners, the same shape as codex-account-store. * fix(windows): drain the ACL flight even when an earlier shutdown release rejects Reviewer P2: the flush sat after backgroundLifecycle/native-main release in one finalizer, so a rejected release skipped it and the EPERM/EBUSY window came back on the failure path. The flush now runs in finally; the original rejection still propagates. Regression test mocks a throwing native release and proves stop() stays pending until the flight settles (driven red). --------- Co-authored-by: jun <[email protected]>
1 parent 7d25f99 commit 19b0157

23 files changed

Lines changed: 416 additions & 69 deletions
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# 000 — Windows CI failure inventory (windows-latest, workflow_dispatch)
2+
3+
Runs read: 33590540220 (ef68bd1f2, cursor stack), 33584155821 (2cb592174, regaudit),
4+
33555110133 (af6113a03 = main). Raw logs in `.tmp/win/*.log` (gitignored). Last Windows-green
5+
dispatch: 33290817128 on 223a0a287 (2026-08-30). Every dispatch since has failed on all branches
6+
including `main`, so this is a regression in the range 223a0a287..dev, not the runner.
7+
8+
## Signatures → owning file → class
9+
10+
| # | Signature | Count | Owner (src) | Owner (tests) | Class |
11+
|---|---|---|---|---|---|
12+
| A | `EPERM rm tests/.tmp-codex-accounts-test` / `.tmp-codex-auth-api-test` | 49 / 377 | `src/config/paths.ts:31-48` (`hardenConfigDir` → unawaited `hardenSecretDirAsync`), `src/lib/windows-secret-acl.ts:345` (icacls spawn) | `tests/codex-account-store.test.ts:37,44,1231,1238`; `tests/codex-auth-api.test.ts:259,296` | product lifecycle regression (e5d588669) + harness cascade |
13+
| B | `EBUSY rm Temp/ocx-*` after `await server.stop(true)` | ~14 suites | `src/server/index.ts:2294-2315` (`server.stop` never awaits config-dir hardening) | kiro-completion:39, claude-native:28, api-usage:103, vision-e2e:45, oauth-live:57, loopback-listener:109, chat-completions:78, pool-mgmt:178, claude-endpoint:67, oauth-accounts-api:64, server-live:37/61, data-plane-admission:232 | same root as A |
14+
| C1 | `EPERM fsync` | 3 + 1 child | `src/lib/service-secrets.ts:68` `fsyncRegularFile` opens `"r"`; same in `src/responses/spill-store.ts:387,425` | `tests/service-secrets.test.ts:93,109,130`; `tests/client-connect.test.ts:494` (rotation child) | product portability defect |
15+
| C2 | icacls `ETIMEDOUT` / access-denied warnings | — | `src/config.ts:2728-2742` | `tests/config.test.ts:2945,2962` inject them | NOT a failure (deliberate test output) |
16+
| D | `Responses previous_response_id state` / `admission boundary` ~45 cases; one 60 s timeout | ~45 | `src/responses/state.ts:1024-1028,1057-1059` (Windows queues async spill) | `tests/responses-state.test.ts:821+, 1045, 1277, 1328, 3190+` | harness: generic cases assume sync lane; :1277 lacks principal-resolver injection and releases its gate outside `finally`, wedging :1328 |
17+
| E | `dev version bump rule` ×3 exit 1 | 3 | — | `tests/bump-dev-version.test.ts:15` uses `new URL(..).pathname` (`/D:/a/...`) | harness |
18+
| F | `config.json JSON Parse error` in ocx-overlay-review | — | — | `tests/user-cost-overlay-coderabbit-regressions.test.ts:89,147` writes `{ not json` on purpose | NOT a failure |
19+
| G | 20-odd single failures (WS handshake, passthrough, readyz, api/usage, Cockpit import, …) | ~20 | — | see B table | all B teardown or A cascade; no independent defect |
20+
21+
## Mechanism (A/B)
22+
23+
loadConfig()/account-store/oauth store → hardenConfigDir()
24+
→ hardenSecretDirAsync() fire-and-forget → icacls.exe holds the dir
25+
→ test/server finishes → rmSync() → EPERM (unlink) / EBUSY (rename)
26+
27+
`server.stop(true)` awaits listeners, background lifecycle and native-main, not this flight.
28+
A flush exists (`src/config/paths.ts:53`) but is test-only. `tests/codex-account-store.test.ts`
29+
stubs `setIcaclsRunnerForTests` only; the async runner (`setAsyncIcaclsRunnerForTests`) stays real.
30+
31+
## Roadmap
32+
33+
- 010 — A/B: production `flushConfigDirHardening(configDir)` awaited in `server.stop`; harness
34+
`removeTreeWithRetry` at every cleanup site; async runner stub in account-store test.
35+
fuck-powershell case: `env-paths/async-child-holds-dir-after-stop`.
36+
- 020 — C1 + D + E: `"r+"` fsync handles; responses-state fixture lanes + gate `finally`;
37+
`fileURLToPath`. fuck-powershell cases: `env-paths/fsync-readonly-handle-eperm`,
38+
`env-paths/file-url-pathname-drive-slash`. Then dispatch CI on an immutable ref → Windows 1-4/4.
39+
- 030 — regression audit main..dev (parallel reviewers), devlog record.
40+
- 040 — promote dev → preview → main via `scripts/release.ts`; proof; bump dev.
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# 010 — Config-dir ACL hardening must be part of the shutdown contract (signatures A, B, G)
2+
3+
Branch `codex/win-acl-lifecycle` off origin/dev.
4+
5+
## src
6+
7+
- `src/config/paths.ts`: promote the test-only flush to `flushConfigDirHardening(configDir: string): Promise<void>`
8+
that awaits the in-flight entry for that exact directory (no-op when none). Keep the existing
9+
test helper as a thin wrapper.
10+
- `src/server/index.ts` `startServer`: capture the effective config dir before `loadConfig()`;
11+
in the composite `server.stop` finalizer (~2312) await `flushConfigDirHardening(dir)` after the
12+
listeners close, before returning.
13+
- Test: `tests/server-stop-config-hardening.test.ts` — `setPlatformForTests("win32")` (otherwise
14+
`windowsSecretAclApplies()` at `windows-secret-acl.ts:469` is false and no flight starts), inject
15+
a controllable async icacls runner via `setAsyncIcaclsRunnerForTests` plus a principal resolver,
16+
start a server, call `stop(true)`, assert it stays pending until the runner settles, then
17+
resolves. Every seam restore and the gate release live in `finally` so an assertion failure
18+
cannot wedge cleanup.
19+
20+
## tests (harness)
21+
22+
Replace recursive `rmSync` with `removeTreeWithRetry` (`tests/helpers/remove-tree.ts`) at:
23+
codex-account-store:37,44,1231,1238; codex-auth-api:259,296; oauth-status-privacy:33,42;
24+
server-kiro-completion-e2e:39; claude-native-passthrough:28; api-usage:103; vision-sidecar-e2e:45;
25+
oauth-login-cli-live-update:57; loopback-listener-integration:109; chat-completions-endpoint:78;
26+
account-pool-management-api:178; claude-messages-endpoint:67; oauth-accounts-api:64;
27+
server-live:37,61; data-plane-admission-identity:232.
28+
29+
`tests/codex-account-store.test.ts`: stub AND restore `setAsyncIcaclsRunnerForTests` alongside the
30+
sync stub; use per-test `mkdtempSync` instead of the fixed repo-local `TEST_DIR` so one failure
31+
cannot poison the rest of the file.
32+
33+
## fuck-powershell
34+
35+
`cases/env-paths/async-child-holds-dir-after-stop.md`: Symptom = rmSync EPERM/EBUSY right after a
36+
clean `server.stop()`; Cause = fire-and-forget `icacls.exe` child + mandatory locking; Workaround =
37+
shutdown contract owns every spawned child; retry-on-EPERM only as a harness fallback.
38+
39+
## Checks
40+
41+
bun test tests/server-stop-config-hardening.test.ts tests/codex-account-store.test.ts tests/codex-auth-api.test.ts tests/remove-tree-helper.test.ts
42+
bun run typecheck
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# 020 — fsync handles, responses-state fixtures, bump CLI path (signatures C1, D, E) + Windows dispatch
2+
3+
Branch `codex/win-fsync-fixtures` on top of 010.
4+
5+
## C1 — read-only fsync (product)
6+
7+
- `src/lib/service-secrets.ts:68` `fsyncRegularFile`: `openSync(path, "r+")`.
8+
- `src/responses/spill-store.ts:387,425`: the copied destination reopened for fsync uses `"r+"`.
9+
- Existing contract test `tests/codex-transition-state-adoption.test.ts:73` already documents the
10+
Windows rule; add one assertion in `tests/service-secrets.test.ts` that the handle mode is
11+
writable (spy `openSync`).
12+
13+
## D — responses-state fixtures (harness)
14+
15+
- Generic spill/admission describe blocks: fixture platform `"linux"` so they exercise the sync
16+
lane; `spill-store.ts::harden` must consult `windowsSecretAclApplies()` consistently.
17+
- Dedicated Windows cases stay `"win32"` and `await flushPendingResponseSpillsForTests()` before
18+
settled-state assertions.
19+
- `tests/responses-state.test.ts:1277`: inject sync+async principal runners like
20+
`tests/helpers/responses-state-never-settling-acl-child.ts:44`; move `release()` into a
21+
`finally` that covers every await after gate creation (this is what wedged :1328 for 60 s).
22+
23+
## E — bump CLI (harness)
24+
25+
`tests/bump-dev-version.test.ts:15`: `fileURLToPath(new URL(...))`, spawn `process.execPath`,
26+
include stderr in the failure message, and make the malformed case assert the specific stderr.
27+
28+
## fuck-powershell
29+
30+
- `cases/env-paths/fsync-readonly-handle-eperm.md`
31+
- `cases/env-paths/file-url-pathname-drive-slash.md`
32+
33+
## Gate
34+
35+
Push both branches, open PRs, then `gh workflow run ci.yml --ref codex/win-dispatch-<sha>` on an
36+
immutable ref of the stacked head. Required: windows 1/4..4/4 SUCCESS. Iterate on failures from
37+
the exact-head logs; never loosen an assertion.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# 030 — Regression audit main..dev
2+
3+
After 010/020 land: `git log --oneline origin/main..origin/dev`, split into 4 lanes (src first half,
4+
src second half, tests-only + gui, security boundary per MAINTAINERS.md) and dispatch read-only
5+
sol/high reviewers in parallel. Each returns VERDICT + per-commit user-impact notes. Record
6+
verbatim tails in `031_verdicts.md`. Any medium+ finding becomes a fix cycle before 040.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# 040 — Promote and bump
2+
3+
Preconditions: dispatch CI green on the dev tip incl. Windows; service-lifecycle.yml green on the
4+
same SHA; regression audit recorded. Read `scripts/release.ts` before running (it accepts only
5+
main/preview and pushes even without --publish). Use a dedicated clean worktree with root + gui
6+
`bun install`. Promote dev → preview (prerelease) → main (stable) per the helper; if a gate fails
7+
after the bump push, rerun once then manual `release.yml` with `expected-sha`.
8+
9+
Proof: `npm view @bitkyc08/opencodex dist-tags --json` + `gitHead`, `gh release view`,
10+
`git ls-remote` for preview/main tips. Then `scripts/bump-dev-version.ts` for the next minor on
11+
dev via PR, admin merge, is-ancestor proof.

‎src/config/paths.ts‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,21 @@ export function hardenConfigDir(): void {
4949
}
5050
}
5151

52-
/** Test-only: settle optional config-directory hardening without exposing it to production callers. */
52+
/**
53+
* Settle the optional hardening flight for one config directory.
54+
*
55+
* The flight spawns `icacls.exe`, which holds the directory open until it exits. Windows file
56+
* locking is mandatory, so anything that removes or renames that directory after a "clean"
57+
* shutdown — a test fixture teardown, an uninstaller, a home move — gets EPERM/EBUSY unless the
58+
* process that started the child also waits for it. `server.stop` calls this so the shutdown
59+
* contract owns every child it started. No-op when nothing is in flight.
60+
*/
61+
export async function flushConfigDirHardening(dir: string = getConfigDir()): Promise<void> {
62+
const flight = configDirHardeningFlights.get(dir);
63+
if (flight) await flight;
64+
}
65+
66+
/** Test-only: settle every in-flight config-directory harden regardless of directory. */
5367
export async function flushConfigDirHardeningForTests(): Promise<void> {
5468
await Promise.all([...configDirHardeningFlights.values()]);
5569
}

‎src/server/index.ts‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import {
2121
websocketsEnabled,
2222
} from "../config";
2323
import { grokDefaultReasoningEffort } from "../grok/effort";
24+
import { flushConfigDirHardening } from "../config/paths";
2425
import { reconcileOAuthProviders } from "../oauth";
2526
import { withCatalogWriteSerialization } from "../codex/catalog-write-serialization";
2627
import { invalidateCodexModelsCacheWithPermit } from "../codex/catalog/sync";
@@ -638,6 +639,9 @@ export function warnAgentTaskRecoveryStartup(config: {
638639

639640
export function startServer(port?: number, deps: StartServerDeps = {}): Server<WsData> {
640641
const localAttestationSecret = deps.localAttestationSecret ?? createLocalAttestationSecret();
642+
// Captured before loadConfig() starts the optional ACL flight so stop() drains the same dir
643+
// even if OPENCODEX_HOME changes underneath a long-lived process.
644+
const startupConfigDir = getConfigDir();
641645
const config = runModelRenameStartupMigration(runAlibabaRegionStartupMigration(runOpenAiTierStartupMigration(loadConfig())));
642646
warnAgentTaskRecoveryStartup(config);
643647
setLiveStateStoreConfig(config);
@@ -2310,8 +2314,16 @@ export function startServer(port?: number, deps: StartServerDeps = {}): Server<W
23102314
},
23112315
],
23122316
async () => {
2313-
await backgroundLifecycle.release();
2314-
await releaseNativeMainStartupLifecycle(server);
2317+
try {
2318+
await backgroundLifecycle.release();
2319+
await releaseNativeMainStartupLifecycle(server);
2320+
} finally {
2321+
// icacls.exe from hardenConfigDir() holds the config dir open; a caller that
2322+
// removes the dir right after stop() settles would hit EPERM/EBUSY on Windows
2323+
// otherwise. Runs even when an earlier release rejected — that rejection still
2324+
// propagates, but not before the child is drained.
2325+
await flushConfigDirHardening(startupConfigDir);
2326+
}
23152327
},
23162328
);
23172329
},

‎tests/account-pool-management-api.test.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import { saveConfig } from "../src/config";
88
import { startServer } from "../src/server";
99
import type { OcxConfig } from "../src/types";
1010
import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home";
11+
import { removeTreeWithRetry } from "./helpers/remove-tree";
1112

1213
function makeCodexConfig(overrides: Partial<OcxConfig> = {}): OcxConfig {
1314
return {
@@ -175,7 +176,7 @@ describe("Anthropic account pool strategy management API", () => {
175176
else process.env.OPENCODEX_HOME = previousHome;
176177
isolatedCodexHome?.restore();
177178
isolatedCodexHome = null;
178-
if (testDir) rmSync(testDir, { recursive: true, force: true });
179+
if (testDir) removeTreeWithRetry(testDir);
179180
});
180181

181182
test("GET /api/oauth/accounts/pool surfaces strategy defaults", async () => {
@@ -496,4 +497,3 @@ describe("generic OAuth pool-settings contract (#695)", () => {
496497
}
497498
});
498499
});
499-

‎tests/api-usage.test.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test";
22
import { managementFetch as fetch } from "./helpers/management-auth";
3-
import { appendFileSync, closeSync, mkdirSync, mkdtempSync, openSync, rmSync, writeFileSync, writeSync } from "node:fs";
3+
import { appendFileSync, closeSync, mkdirSync, mkdtempSync, openSync, writeFileSync, writeSync } from "node:fs";
44
import { tmpdir } from "node:os";
55
import { join } from "node:path";
66
import { saveConfig } from "../src/config";
@@ -9,6 +9,7 @@ import type { OcxConfig } from "../src/types";
99
import { refreshUserCostOverlays, resetPreservedDiskOnlyProvidersForTests, userCostOverlayVersion } from "../src/usage/user-cost-overlays";
1010
import { stopUserCostOverlayReconciler } from "../src/usage/user-cost-overlay-reconciler";
1111
import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home";
12+
import { removeTreeWithRetry } from "./helpers/remove-tree";
1213
import { resetUsageReadCacheForTests, setManagementUsageMaxEntriesForTests, usageReadCacheStatsForTests } from "../src/usage/log";
1314
import * as usageLogModule from "../src/usage/log";
1415
import { getUsageSummaryCacheEntry, resetUsageSummaryCacheForTests } from "../src/server/management/usage-summary-cache";
@@ -100,7 +101,7 @@ afterEach(() => {
100101
else process.env.OPENCODEX_HOME = previousHome;
101102
isolatedCodexHome?.restore();
102103
isolatedCodexHome = null;
103-
if (testDir) rmSync(testDir, { recursive: true, force: true });
104+
if (testDir) removeTreeWithRetry(testDir);
104105
});
105106

106107
describe("GET /api/usage", () => {

‎tests/chat-completions-endpoint.test.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
import { waitForNativeMainStartupGate } from "../src/codex/native-profile-startup";
22
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
3-
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
3+
import { mkdtempSync, writeFileSync } from "node:fs";
44
import { tmpdir } from "node:os";
55
import { join } from "node:path";
66
import { saveConfig } from "../src/config";
77
import { startServer } from "../src/server";
88
import { ownedServiceHomeInspection } from "./helpers/owned-service-home-inspection";
99
import type { OcxConfig, OcxProviderConfig } from "../src/types";
1010
import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home";
11+
import { removeTreeWithRetry } from "./helpers/remove-tree";
1112
import { chatCompletionsToResponsesBody, ChatCompletionsRequestError } from "../src/chat/inbound";
1213
import { chatCompletionsUsage } from "../src/chat/outbound";
1314
import { parseRequest } from "../src/responses/parser";
@@ -75,7 +76,7 @@ afterEach(() => {
7576
isolatedCodexHome?.restore();
7677
isolatedCodexHome = null;
7778
globalThis.fetch = originalFetch;
78-
if (testDir) rmSync(testDir, { recursive: true, force: true });
79+
if (testDir) removeTreeWithRetry(testDir);
7980
});
8081

8182
function mockChatUpstream() {

0 commit comments

Comments
 (0)