Skip to content

Commit ca32042

Browse files
authored
Merge pull request #1949 from lidge-jun/codex/windows-stability-program
devlog: open the Windows stability program unit
2 parents ac39093 + 5ca4ffe commit ca32042

15 files changed

Lines changed: 1224 additions & 0 deletions
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# 000 — Windows stability: why "806/806 green" is not "stable"
2+
3+
Unit opened 2026-08-17, after v2.24.2 shipped.
4+
5+
## The gap this unit exists to close
6+
7+
The Windows campaign that preceded this unit took the local Bun suite from 53+
8+
failures to 806/806 across 15 commits. That was real work on real defects — an
9+
empty-string `LocalApplicationData`, unfinalized SQLite statements holding a
10+
file open against unlink, TOML escapes doubling backslashes, per-process
11+
identity lookups costing ~510ms each.
12+
13+
None of it proves the product is stable for a Windows user, and the reason is
14+
structural rather than rhetorical: **the suite that went green is not a gate.**
15+
16+
```yaml
17+
# .github/workflows/ci.yml:547-552
18+
platform-windows:
19+
name: windows ${{ matrix.shard }}/4
20+
needs: select-windows-runner
21+
if: github.event_name == 'workflow_dispatch'
22+
```
23+
24+
Windows runs only when a maintainer asks by hand. The aggregation job at
25+
`.github/workflows/ci.yml:747-783` accepts `skipped` as an outcome, and
26+
`.github/workflows/release.yml:181-201` requires a successful **push-event**
27+
CI run before publishing. Since `platform-windows` always skips on push, a
28+
release satisfies its own gate having executed zero Windows tests.
29+
30+
Issue #1059 tracks exactly this and is still open. Its stated end condition is
31+
Windows restored as a required gate. The failure counts quoted there are now
32+
stale in our favour; the workflow contract has not caught up.
33+
34+
## Evidence base for this unit
35+
36+
Three independent GPT-5 Pro audits were run on 2026-08-17 against a zip of the
37+
v2.24.2 tree (`src/`, `tests/`, `scripts/`, `.github/`, `structure/`), each
38+
with the GitHub connector attached and a distinct brief:
39+
40+
| Chat | Perspective | Conversation |
41+
|---|---|---|
42+
| P1 | Platform primitives: handles, locking, atomic publication, paths, ACLs | `chatgpt.com/c/6a82ebc4-48d4-83ee-a223-a6fc5a9556e5` |
43+
| P2 | Runtime and distribution: install, spawn, service lifecycle, update, ports | `chatgpt.com/c/6a82ec28-86b4-83e8-86e4-a5477b6a9d91` |
44+
| P3 | User-visible failure modes, diagnostics, and CI coverage | `chatgpt.com/c/6a82ec41-6b0c-83ee-93ab-3a96010a543f` |
45+
46+
Every finding carried into `001` was **re-verified against the working tree in
47+
this session**. Claims that could not be reproduced locally were dropped rather
48+
than recorded. That rule matters here because two of the three audits also
49+
correctly identified defects as *already fixed* (#1843 elevation argv, #31
50+
passthrough segfault) — an audit that cannot tell live from historical is not
51+
usable as a roadmap input.
52+
53+
## What changed in the problem model
54+
55+
The pre-campaign model was "Windows has many small filesystem bugs." The
56+
evidence no longer supports that as the dominant class. The surviving defects
57+
cluster into three shapes:
58+
59+
1. **Synchronous Windows subprocesses on the request path.** `icacls` and
60+
PowerShell/CIM calls that block Bun's event loop. This is invisible to a
61+
test suite that never measures latency under concurrency.
62+
2. **Lifecycle operations that are not transactional.** Update and native
63+
service migration both destroy working state before proving the replacement.
64+
3. **Invariants enforced by prose or by a single-file test, so they drift.**
65+
The `-WindowStyle Hidden` case in `001` is the clearest example.
66+
67+
None of those three are things a per-file unit test naturally catches, which is
68+
why 806 green files and an unhappy user base are consistent with each other.
Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
# 001 — Verified findings
2+
3+
Every entry below was reproduced against the working tree at `474584bcd` on
4+
2026-08-17. Line numbers are from that tree. Findings the audits raised that
5+
could not be reproduced are listed at the bottom under "Not carried".
6+
7+
Ranked by user impact.
8+
9+
---
10+
11+
## F1 — `src/service.ts:2361` uses the exact PowerShell argv the codebase forbids
12+
13+
`killWindowsServiceWrapperProcesses()` in `src/service.ts` spawns:
14+
15+
```ts
16+
// src/service.ts:2360-2363
17+
spawnSync(resolveTrustedWindowsPowerShellExe(), [
18+
"-NoProfile", "-NoLogo", "-NonInteractive", "-WindowStyle", "Hidden",
19+
"-Command", ps,
20+
], { stdio: "ignore", timeout: 5000, windowsHide: true });
21+
```
22+
23+
The codebase already knows this is wrong. `src/codex/user-identity.ts:222-224`:
24+
25+
> Do not add PowerShell's `-WindowStyle Hidden` here: Bun 1.3.14 can fail that
26+
> direct CLI combination before the SID command executes (#1589); the
27+
> process-level `windowsHide` flag is sufficient.
28+
29+
**Why it survived.** The regression test is scoped to one file:
30+
31+
```ts
32+
// tests/windows-deploy-close-regressions.test.ts:43
33+
expect(src).not.toContain('["-NoProfile", "-NoLogo", "-NonInteractive", "-WindowStyle", "Hidden", "-Command", ps]');
34+
```
35+
36+
`src` there is `read("src/update/job.ts")` (line 13). `src/service.ts` is never
37+
checked. A search of `src/` finds exactly one surviving production occurrence
38+
of that CLI pair: `src/service.ts:2361`.
39+
40+
**User-visible consequence.** `stopServiceIfInstalled()` calls this function
41+
because `schtasks /end` can leave the `wscript.exe`/`cmd.exe` wrapper alive,
42+
which then respawns the proxy. The call ignores `spawnSync`'s exit status and
43+
swallows errors, so under #1589 wrapper termination silently does nothing:
44+
`ocx stop`, restart, and update appear to succeed and do not stick.
45+
46+
Severity: high. Fix cost: one line. Phase 010.
47+
48+
---
49+
50+
## F2 — The wrapper killer exists twice and the copies have drifted apart
51+
52+
Two implementations of the same operation:
53+
54+
```ts
55+
// src/service.ts:2337-2358 — canonical token matching scoped to THIS home
56+
// (paths built 2340-2341; token boundaries enforced 2350-2355)
57+
// src/update/job.ts:1377-1383 (the bare -like match is line 1383)
58+
"$pats = @('opencodex-service.cmd','opencodex-service-launcher.vbs');"
59+
...
60+
"foreach ($p in $pats) { if ($c -like ('*' + $p + '*')) { return $true } };"
61+
```
62+
63+
The updater copy matches a bare filename anywhere in a command line. Two
64+
OpenCodex homes under one Windows account means a dashboard update for home A
65+
can terminate home B's scheduler wrapper. Any unrelated process whose command
66+
line contains either filename also matches.
67+
68+
Cited precisely: the updater's bare match is `src/update/job.ts:1383`; the service copy builds canonical paths at `src/service.ts:2340-2341` and enforces token boundaries at `:2350-2355`.
69+
70+
The drift is already measurable and runs in both directions: `update/job.ts`
71+
received the #1589 argv cleanup that `service.ts` missed (F1); `service.ts`
72+
received canonical path scoping that `update/job.ts` missed. Two copies, two
73+
different half-fixes.
74+
75+
Severity: high (cross-installation process kill). Phase 020.
76+
77+
---
78+
79+
## F3 — Windows is not a gate, and the release gate cannot see that
80+
81+
```yaml
82+
# .github/workflows/ci.yml:547-552
83+
if: github.event_name == 'workflow_dispatch'
84+
```
85+
86+
The aggregation job accepts `skipped` (`ci.yml:769-772` — the jq filter keeps
87+
only jobs that are neither `success` nor `skipped`). The release preflight
88+
(`release.yml:181-201`) demands a successful **push-event** `ci.yml` run —
89+
deliberately narrower than "any successful run for this SHA" — but
90+
`platform-windows` never runs on push. So the general release preflight does not require `platform-windows`, and a
91+
release can publish without it having run.
92+
93+
One qualification, because the stronger claim is not true: releases that touch
94+
`src/service.ts`, `src/cli/index.ts`, `package.json` and a few others separately
95+
require a green `service-lifecycle.yml` (`release.yml:224-241`, enforced at
96+
235-239), and that
97+
workflow does include a Windows job. Windows is therefore not entirely absent
98+
from release gating - it is absent from the *suite* gate, and present only as a
99+
lifecycle smoke test for service-shaped changes.
100+
101+
Severity: high, and it is the multiplier on every other finding — without it,
102+
each fix below is one careless merge away from regressing. Phases 060 and 070.
103+
104+
---
105+
106+
## F4 — Durable publishers do not share the Windows retry primitive
107+
108+
`src/config.ts:102-123` knows about Windows sharing violations:
109+
110+
```ts
111+
const transientWindowsError = io.platform === "win32"
112+
&& (code === "EBUSY" || code === "EPERM" || code === "EACCES");
113+
if (!transientWindowsError || attempt >= 2) throw error;
114+
io.sleep(25 * (attempt + 1));
115+
```
116+
117+
Two retries, 25ms then 50ms: about 75ms of total tolerance. Other durable
118+
publishers do not call it at all and use raw `renameSync`:
119+
120+
- `src/codex/prompt-journal.ts` — publishes a journal holding full
121+
`config.toml` bytes
122+
- `src/lib/config-ownership.ts` — publishes the uninstall ownership manifest
123+
124+
These are fail-safe, not corrupting: they throw rather than publish a partial
125+
file. But under a real-time scanner or a sync client holding the target, they
126+
turn a recoverable hiccup into a user-visible operational failure.
127+
128+
The 75ms envelope is itself a watch item, not yet a defect — we have no field
129+
telemetry showing Defender or OneDrive holding files longer. Instrument before
130+
widening. Phase 030 makes the primitive shared; Phase 031 adds the counters.
131+
132+
---
133+
134+
## F5 — `chmod` is load-bearing where it does nothing
135+
136+
`src/config.ts` calls `chmodSync(target, 0o600)` at lines 221, 316, 450, 1713,
137+
2683 and 3942, and `chmodSync(dir, 0o700)` at 1704, 2632, each wrapped in
138+
`catch { /* platform may ignore chmod */ }`. The 3942 site sits inside
139+
`backupInvalidConfig` (declared at 3937), which copies the whole config
140+
including whatever secrets it held. On Windows the call is a no-op:
141+
the ACL is what protects the file, and `src/lib/windows-secret-acl.ts` is what
142+
sets it.
143+
144+
Where both run, the file is protected. The audit work needed here is an
145+
inventory: every path that writes a credential, token, or OAuth refresh token,
146+
and whether the Windows ACL path is reached on that specific write or only the
147+
`chmod`. `src/service.ts:1983` states the ACL is authoritative, but says so about an
148+
elevation staging directory specifically. That is evidence for the principle,
149+
not evidence about any credential writer's coverage - each inventory row needs
150+
its own citation.
151+
152+
Treated as **unproven** until the inventory is done. Phase 040. Per AGENTS.md,
153+
if that inventory turns up a live exposure the writeup goes to scratch space,
154+
not into this directory.
155+
156+
---
157+
158+
## F6 — The service wrapper retries a deterministic crash forever
159+
160+
```bat
161+
:: src/service.ts:1556-1563
162+
"%OCX_BUN%" "%OCX_CLI%" start ...
163+
if %ERRORLEVEL% NEQ 0 (
164+
... restarting in 5s
165+
ping -n 6 127.0.0.1 >nul
166+
goto loop
167+
)
168+
```
169+
170+
A proxy that starts successfully and then crashes deterministically is
171+
relaunched every five seconds indefinitely. #1877 deliberately fixed only the
172+
missing-executable case, on the reasoning that a flat "N failures then stop"
173+
ceiling would break recovery from intermittent faults. That reasoning is sound;
174+
the conclusion does not have to be an unbounded fixed-interval loop.
175+
176+
Capped exponential backoff with a health-reset — 5s, 15s, 30s, 60s, reset after
177+
sustained uptime — preserves recovery and stops the log storm. Phase 050.
178+
179+
---
180+
181+
## F7 — Windows CI never proves crash-restart
182+
183+
`.github/workflows/service-lifecycle.yml:104-135` kills the systemd MainPID,
184+
waits for a different PID, and asserts `/healthz`. The Windows job
185+
(`windows-schtasks`, line 239) only covers install, health, clean `ocx stop`,
186+
uninstall. The restart path F6 describes has no coverage on the platform where
187+
it is implemented in batch. Phase 051.
188+
189+
---
190+
191+
## Not carried
192+
193+
Raised by the audits, deliberately excluded:
194+
195+
- **#1843 elevated `Start-Process` argv** — already fixed; PR #1860 merged and
196+
present in the tree.
197+
- **#31 passthrough SSE segfault** — fixed via `body.tee()`.
198+
- **Bun replacing its own running executable during update** —
199+
`src/update/index.ts:152-155` documents that the plain-Node launcher handles
200+
npm self-update before Bun starts.
201+
- **Synchronous `icacls`/CIM on the request path (#1852, #1298; PR #1876)** —
202+
both P1 and P3 rate this their top runtime issue and the reasoning is
203+
persuasive, but it is a latency property this session did not measure. It
204+
belongs to the open PR, not to this unit. Recorded here so the next cycle
205+
starts from it rather than rediscovering it.
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# 002 — Sequencing and what this unit deliberately does not do
2+
3+
The first draft of this document claimed a long dependency chain. A plan audit
4+
(round `r1-20260817113441`) showed most of it was file-overlap dressed up as
5+
dependency, and one link was backwards. This is the corrected version; the
6+
reasoning is kept because "why we thought these were dependencies" is the more
7+
useful record.
8+
9+
## Real dependencies
10+
11+
Only two links are structural:
12+
13+
```mermaid
14+
graph LR
15+
A["030 shared replace primitive"] --> B["031 retry telemetry"]
16+
C["060 stage 1 - run non-gating"] --> D["070 flakiness policy"]
17+
```
18+
19+
`030 → 031` because there is nothing to instrument until the primitive exists.
20+
`060 stage 1 → 070` because the flakiness policy is calibrated on the failure
21+
data stage 1 produces.
22+
23+
Everything else is schedulable now.
24+
25+
## Start immediately, in parallel
26+
27+
- **060 stage 1** — highest priority despite its number. It only makes Windows
28+
*run*; it blocks nothing, and every later phase wants its data. Its one
29+
prerequisite is the runner-policy decision inside 060, which is a decision to
30+
make rather than work to schedule.
31+
- **010** — one line plus a widened guard.
32+
- **051** — crash-restart already exists, so it is testable today. Landing it
33+
before 050 gives the timing change a baseline.
34+
- **040** — independent inventory, produces a document.
35+
36+
## Ordering preferences that are not dependencies
37+
38+
Stated so nobody mistakes them for blockers:
39+
40+
- **010 before 020** was originally justified as "otherwise the fix is written
41+
twice". That is wrong: deduplicating first moves one flawed implementation,
42+
and 010 then fixes it once. Either order works. Prefer 010 first only because
43+
it is trivial and unblocks nothing else.
44+
- **020 before 030** is people-not-colliding in `service.ts` and `job.ts`.
45+
- **010/020 before 050** is the same, all three touch `src/service.ts`.
46+
- **050 before 051** was fake, and worse, it produced an impossible verification
47+
claim — 051 now says plainly that it cannot verify 050's backoff.
48+
- **"everything before 060"** was false. None of F1, F2, F4, F5 or F6 makes the
49+
suite red today. What is true is narrower: **060 stages 3 and 4** should wait
50+
for the fixes, because that is when a Windows failure starts costing someone
51+
a merge or a release.
52+
- **080** is simplest to add once 060 stage 1 has a Windows leg running, but it
53+
is not blocked by it; it starts non-gating and does not wait for stage 3.
54+
55+
Each phase header states its own dependency line. Where a header says "sequence
56+
around" another phase, that is collision avoidance in shared files — `002` is
57+
authoritative on what is structural, and only the two links above are.
58+
59+
## Out of scope for this unit
60+
61+
**The synchronous-subprocess latency class.** Both P1 and P3 rank
62+
`icacls`/PowerShell-CIM on the request path as the top runtime problem
63+
(#1852, #1298, PR #1876). It is excluded because this session measured nothing —
64+
no latency numbers, no event-loop traces. Carrying it would put an unverified
65+
claim beside seven verified ones and devalue all of them.
66+
67+
The audit accepted that exclusion as honest and then made the sharper point:
68+
because `000` itself names this the leading runtime class, finishing this unit
69+
**cannot** establish "Windows is stable". It establishes a reliability and CI
70+
baseline while the highest-ranked risk stays open in #1876. That is the accurate
71+
claim and the one to make in any release note.
72+
73+
**Update transactionality.** #1849 is open and the design work — stage outside
74+
the live tree, verify, switch, retire the backup — is larger than any phase
75+
here. Separate unit.
76+
77+
**Branch protection.** 060 cannot make Windows block a merge; `dev` has no
78+
protection and `MAINTAINERS.md:121` and `:125` record that enforcing anything
79+
that way is an unmade decision. Configuring it is a maintainer call, not a
80+
phase.
81+
82+
## Definition of done for the unit
83+
84+
- 010, 020, 030, 031, 050, 051 landed, each guard driven red before it counts.
85+
- 060 through stage 4, so a release preflight cannot pass on a push run where
86+
Windows silently skipped.
87+
- 060's runner policy explicitly resolved rather than left implicit.
88+
- 070's nightly running, quarantine list open and reviewed each release.
89+
- 080 items 1-5 landed; items 6 and 7 landed or documented as not achievable.
90+
- 040's inventory complete, with any live exposure handled entirely in scratch
91+
per AGENTS.md and nothing about it written here.
92+
93+
Until 060 stage 4 is done, every fix in this unit is one careless merge from
94+
regressing. That is the point of the unit.

0 commit comments

Comments
 (0)