Skip to content

Commit a9590de

Browse files
committed
docs(structure): memory watchdog SOT — design rationale, platform scope, audit trail
Permanent record for the PR: why the watchdog exists (79GB committed / 97% system commit field incident), the async-collect/sync-decide architecture and its defended invariants, the platform matrix (cross-platform core, Windows- first measurement fidelity; v1 system-commit axis is Windows-only), why the commit axis is observe-only in v1 (it cannot know the pressure is ours — and honestly does NOT auto-mitigate the original incident yet), the quiet-window restart + honestly-scoped loop guards, verification evidence incl. live measurements, the three audit rounds, and deferred follow-ups.
1 parent 9942240 commit a9590de

1 file changed

Lines changed: 129 additions & 0 deletions

File tree

‎structure/09_memory-watchdog.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Memory Watchdog SOT
2+
3+
## Why this exists
4+
5+
Field incident: OpenCodex on Bun reached **~79 GB private/committed memory while the working set
6+
stayed ~5.8 GB**, freezing the box once the **system commit charge hit ~97% of the commit limit**.
7+
The Bun/mimalloc native allocator retains committed memory the JS heap has already released
8+
(mirrored upstream by Claude Code #36132); no in-app JS cache cap can fix native retention. The
9+
locally-run stress harness reproduces the detectability signature on a healthy machine
10+
(`afterClear private/RSS = 4.4–4.8` after clearing the JS store and forcing GC).
11+
12+
The only proven mitigation is to **observe the pressure and, opt-in, hand off to a graceful
13+
restart**. Everything below follows from that.
14+
15+
## Platform scope (PR reviewers: read this first)
16+
17+
The watchdog is **cross-platform**; only the measurement *fidelity* differs per platform. This is
18+
NOT a Windows-only patch, but the motivating incident, the highest-fidelity probe, and the v1
19+
system-commit axis are Windows-specific.
20+
21+
| Layer | win32 | linux | other |
22+
| --- | --- | --- | --- |
23+
| Process pressure source | Private Bytes (async PowerShell probe) | VmRSS+VmSwap (`/proc/self/status`, no spawn) | RSS (in-process) |
24+
| System commit axis (v1) | measured (CommittedBytes/CommitLimit) | not measured (`systemCommitAvailable=false`; `/proc/meminfo` `Committed_AS` is a documented follow-up) | not measured |
25+
| Decision core (warn/critical, cooldown, max-restarts, history seeding) | identical | identical | identical |
26+
| Quiet-window drain + exit-75 restart, supervisor gating | identical | identical | identical |
27+
| Dashboard / management API / i18n | identical | identical | identical |
28+
29+
The commit-axis logic itself is platform-neutral and null-safe: on platforms without measurement
30+
it simply skips (no NaN propagation, latch held), so enabling a Linux collector later is additive.
31+
32+
## Architecture: async collect, sync decide
33+
34+
```text
35+
self-rescheduling probe loop (unref'd; first probe fires IMMEDIATELY at start)
36+
└─ captureMemorySnapshot() — async, never throws, never blocks the event loop
37+
win32: ONE hidden PowerShell child (array args, -NoProfile -NonInteractive) returns
38+
labeled values P=/C=/L=/A= (private, committed, commit limit, available physical;
39+
Win32_PerfRawData_PerfOS_Memory — all bytes, verified live). 15s budget
40+
(measured cold spawn: 4.0–5.9s); timeout kills the child; first-resolution-wins
41+
drops late results; failures degrade to an RSS fallback with a sanitized probeError.
42+
└─ tick(state, cfg, deps, snapshot) — sync, pure-core evaluate(), EXACTLY once per probe
43+
axis 1 PROCESS: pressure/physicalRAM vs warn(0.60)/critical(0.75) → opt-in restart path
44+
axis 2 SYSTEM COMMIT: observe-only high-water warning (see below), NEVER restarts
45+
└─ report cache (memory block, responseStateMetrics, lastProbeAt/lastSuccessfulSystemProbeAt)
46+
computed once per probe — a 5s dashboard poll never re-serializes the response store
47+
```
48+
49+
Key invariants, each defended by a regression test:
50+
51+
- **The probe never blocks the event loop.** The original implementation used `Bun.spawnSync`
52+
(up to 2s synchronous stall per tick on Windows) — flagged as a deploy blocker by audit and
53+
replaced. A streaming proxy must not pause SSE for measurement.
54+
- **Exactly one evaluation per probe.** Timeout → RSS-fallback evaluation, and a late success for
55+
the same probe is dropped; an unexpected `capture()` exception evaluates once on
56+
`rssFallbackSnapshot(..., "capture-threw")` instead of silencing the cycle.
57+
- **A restart owns the exit.** `tick()`'s restart hook synchronously stops the watchdog;
58+
`probeOnce` pins its instance in a local and re-checks IDENTITY against the global after
59+
capture, after tick, and before rescheduling, so the stopped/replaced singleton is never
60+
touched or rescheduled (pre-fix this was an unhandled TypeError on every real auto-restart).
61+
- **Stop kills the child.** `stopMemoryWatchdog()` aborts the in-flight capture via
62+
AbortController and a generation guard drops any result that still arrives.
63+
64+
## System-commit axis: why observe-only (v1)
65+
66+
`systemCommittedBytes / systemCommitLimitBytes >= 0.90` (env-only override
67+
`OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER`, clamped [0.50, 0.99]) logs ONE latched warning; the latch
68+
re-arms only on a MEASURED recovery — measurement loss (probe failure) holds it, so a flapping
69+
probe cannot re-warn each time it returns.
70+
71+
It never contributes to the restart decision because the commit pressure may come from **another
72+
process**: restarting OpenCodex would not free it, and an automated restart loop against an
73+
external cause is worse than a loud warning. Honest consequence, stated up front: **v1 does not
74+
auto-mitigate the original incident** (79 GB / 128 GB RAM ≈ 62% process pressure < 0.75 critical;
75+
system commit 97% → warning only). Promoting the commit axis into the restart decision, and any
76+
UI-configurable thresholds, are deliberately deferred until field measurement validates the axis.
77+
78+
## Restart path (quiet-window) and honest loop guards
79+
80+
- `restartGraceMs` (default 30s, clamped [1s, 10min] on every entry path — env, config file,
81+
management API which 400s out-of-range values): draining rejects new turns with 503+Retry-After
82+
and returns the moment the in-flight set empties, so the restart lands on a natural idle gap and
83+
the value is a DEADLINE, not a delay.
84+
- `minRestartIntervalMs` is normalized to ≥ `restartGraceMs` so a second restart can never arm
85+
inside the first drain window.
86+
- Cooldown/max-restarts counters die with the process a restart ends, so they are re-seeded from
87+
a best-effort timestamps-only history file (rolling ~6h window, atomic-rename write, every
88+
failure swallowed). This is **not** a permanent cross-process guarantee: the supervisor's own
89+
restart limit/backoff (pm2 `max_restarts`, systemd `StartLimitBurst`) remains the outer layer,
90+
and exit code 75 is only a *request* to respawn. NSSM / Windows services are not auto-detected —
91+
set `OCX_SUPERVISED=1` there.
92+
93+
## Verification evidence (as merged)
94+
95+
- Unit/integration: `tests/memory-watchdog.test.ts` (66), `tests/memory-restart-history.test.ts`
96+
(11), `tests/memory-api.test.ts` (16) — includes a REAL Windows end-to-end probe test asserting
97+
private bytes and a sane commit fraction from the live labeled command.
98+
- Live measurements on a Windows 11 box: probe cold spawn 4.0–5.9s (hence the 15s async budget);
99+
`Win32_PerfRawData_PerfOS_Memory` values confirmed to be bytes; harness detectability
100+
`afterClear private/RSS = 4.4–4.8`.
101+
- Gates: `typecheck`, `lint:gui`, `build:gui`, `privacy:scan` all exit 0. `probeError` carries
102+
sanitized codes only (never raw command output/paths) — privacy-scan enforced.
103+
- Full-suite caveat: on the development Windows box the suite exits 1 due to pre-existing
104+
environment failures (NTFS ACL hardening `EACCES` clusters in auth tests + branch-dependent
105+
release-helper tests). Evidence that these are unrelated: the failure set at this branch's HEAD
106+
is a strict subset (321 ⊂ 345, zero new files, zero memory-file failures) of the same run at the
107+
pre-change baseline commit under identical conditions.
108+
109+
## Audit trail
110+
111+
Three independent audit rounds shaped this design (summarized honestly rather than re-litigated):
112+
113+
1. Initial audit (deploy blockers): synchronous `spawnSync` probe stall, missing system-commit
114+
measurement, 60s startup blind spot, RSS-only stress assertions, unwired response-state
115+
metrics → all five confirmed against code and fixed by the async rewrite.
116+
2. Design review: chose "cached-free" probe-completion-driven evaluation over defer-until-idle
117+
and blue-green alternatives (complexity/regression cost vs. benefit), kept warn-only defaults.
118+
3. Post-implementation audit: auto-restart race in `probeOnce` (unhandled TypeError), capture-throw
119+
silence, positional PowerShell parsing, live re-clamp gaps → all four fixed with regression
120+
tests (the race test was validated to fail against the pre-fix code).
121+
122+
## Follow-ups (out of scope here)
123+
124+
- Field measurement, then possible commit-axis restart integration + UI thresholds (v2).
125+
- Linux system commit via `/proc/meminfo` (`Committed_AS`/`CommitLimit`, no spawn).
126+
- docs-site ja/zh-cn translations of the memoryWatchdog reference section.
127+
- Separate issue: `windows-secret-acl` hardening is non-atomic under icacls timeouts
128+
(inheritance stripped before grant), which is the root cause of the environmental `EACCES`
129+
test failures on slow-ACL machines.

0 commit comments

Comments
 (0)