|
| 1 | +# Memory Watchdog SOT |
| 2 | + |
| 3 | +## Why this exists |
| 4 | + |
| 5 | +Field incident: OpenCodex on Bun reached **~79 GB private/committed memory while the working set |
| 6 | +stayed ~5.8 GB**, freezing the box once the **system commit charge hit ~97% of the commit limit**. |
| 7 | +The Bun/mimalloc native allocator retains committed memory the JS heap has already released |
| 8 | +(mirrored upstream by Claude Code #36132); no in-app JS cache cap can fix native retention. The |
| 9 | +locally-run stress harness reproduces the detectability signature on a healthy machine |
| 10 | +(`afterClear private/RSS = 4.4–4.8` after clearing the JS store and forcing GC). |
| 11 | + |
| 12 | +The only proven mitigation is to **observe the pressure and, opt-in, hand off to a graceful |
| 13 | +restart**. Everything below follows from that. |
| 14 | + |
| 15 | +## Platform scope (PR reviewers: read this first) |
| 16 | + |
| 17 | +The watchdog is **cross-platform**; only the measurement *fidelity* differs per platform. This is |
| 18 | +NOT a Windows-only patch, but the motivating incident, the highest-fidelity probe, and the v1 |
| 19 | +system-commit axis are Windows-specific. |
| 20 | + |
| 21 | +| Layer | win32 | linux | other | |
| 22 | +| --- | --- | --- | --- | |
| 23 | +| Process pressure source | Private Bytes (async PowerShell probe) | VmRSS+VmSwap (`/proc/self/status`, no spawn) | RSS (in-process) | |
| 24 | +| System commit axis (v1) | measured (CommittedBytes/CommitLimit) | not measured (`systemCommitAvailable=false`; `/proc/meminfo` `Committed_AS` is a documented follow-up) | not measured | |
| 25 | +| Decision core (warn/critical, cooldown, max-restarts, history seeding) | identical | identical | identical | |
| 26 | +| Quiet-window drain + exit-75 restart, supervisor gating | identical | identical | identical | |
| 27 | +| Dashboard / management API / i18n | identical | identical | identical | |
| 28 | + |
| 29 | +The commit-axis logic itself is platform-neutral and null-safe: on platforms without measurement |
| 30 | +it simply skips (no NaN propagation, latch held), so enabling a Linux collector later is additive. |
| 31 | + |
| 32 | +## Architecture: async collect, sync decide |
| 33 | + |
| 34 | +```text |
| 35 | +self-rescheduling probe loop (unref'd; first probe fires IMMEDIATELY at start) |
| 36 | + └─ captureMemorySnapshot() — async, never throws, never blocks the event loop |
| 37 | + win32: ONE hidden PowerShell child (array args, -NoProfile -NonInteractive) returns |
| 38 | + labeled values P=/C=/L=/A= (private, committed, commit limit, available physical; |
| 39 | + Win32_PerfRawData_PerfOS_Memory — all bytes, verified live). 15s budget |
| 40 | + (measured cold spawn: 4.0–5.9s); timeout kills the child; first-resolution-wins |
| 41 | + drops late results; failures degrade to an RSS fallback with a sanitized probeError. |
| 42 | + └─ tick(state, cfg, deps, snapshot) — sync, pure-core evaluate(), EXACTLY once per probe |
| 43 | + axis 1 PROCESS: pressure/physicalRAM vs warn(0.60)/critical(0.75) → opt-in restart path |
| 44 | + axis 2 SYSTEM COMMIT: observe-only high-water warning (see below), NEVER restarts |
| 45 | + └─ report cache (memory block, responseStateMetrics, lastProbeAt/lastSuccessfulSystemProbeAt) |
| 46 | + computed once per probe — a 5s dashboard poll never re-serializes the response store |
| 47 | +``` |
| 48 | + |
| 49 | +Key invariants, each defended by a regression test: |
| 50 | + |
| 51 | +- **The probe never blocks the event loop.** The original implementation used `Bun.spawnSync` |
| 52 | + (up to 2s synchronous stall per tick on Windows) — flagged as a deploy blocker by audit and |
| 53 | + replaced. A streaming proxy must not pause SSE for measurement. |
| 54 | +- **Exactly one evaluation per probe.** Timeout → RSS-fallback evaluation, and a late success for |
| 55 | + the same probe is dropped; an unexpected `capture()` exception evaluates once on |
| 56 | + `rssFallbackSnapshot(..., "capture-threw")` instead of silencing the cycle. |
| 57 | +- **A restart owns the exit.** `tick()`'s restart hook synchronously stops the watchdog; |
| 58 | + `probeOnce` pins its instance in a local and re-checks IDENTITY against the global after |
| 59 | + capture, after tick, and before rescheduling, so the stopped/replaced singleton is never |
| 60 | + touched or rescheduled (pre-fix this was an unhandled TypeError on every real auto-restart). |
| 61 | +- **Stop kills the child.** `stopMemoryWatchdog()` aborts the in-flight capture via |
| 62 | + AbortController and a generation guard drops any result that still arrives. |
| 63 | + |
| 64 | +## System-commit axis: why observe-only (v1) |
| 65 | + |
| 66 | +`systemCommittedBytes / systemCommitLimitBytes >= 0.90` (env-only override |
| 67 | +`OCX_MEMORY_WATCHDOG_COMMIT_HIGH_WATER`, clamped [0.50, 0.99]) logs ONE latched warning; the latch |
| 68 | +re-arms only on a MEASURED recovery — measurement loss (probe failure) holds it, so a flapping |
| 69 | +probe cannot re-warn each time it returns. |
| 70 | + |
| 71 | +It never contributes to the restart decision because the commit pressure may come from **another |
| 72 | +process**: restarting OpenCodex would not free it, and an automated restart loop against an |
| 73 | +external cause is worse than a loud warning. Honest consequence, stated up front: **v1 does not |
| 74 | +auto-mitigate the original incident** (79 GB / 128 GB RAM ≈ 62% process pressure < 0.75 critical; |
| 75 | +system commit 97% → warning only). Promoting the commit axis into the restart decision, and any |
| 76 | +UI-configurable thresholds, are deliberately deferred until field measurement validates the axis. |
| 77 | + |
| 78 | +## Restart path (quiet-window) and honest loop guards |
| 79 | + |
| 80 | +- `restartGraceMs` (default 30s, clamped [1s, 10min] on every entry path — env, config file, |
| 81 | + management API which 400s out-of-range values): draining rejects new turns with 503+Retry-After |
| 82 | + and returns the moment the in-flight set empties, so the restart lands on a natural idle gap and |
| 83 | + the value is a DEADLINE, not a delay. |
| 84 | +- `minRestartIntervalMs` is normalized to ≥ `restartGraceMs` so a second restart can never arm |
| 85 | + inside the first drain window. |
| 86 | +- Cooldown/max-restarts counters die with the process a restart ends, so they are re-seeded from |
| 87 | + a best-effort timestamps-only history file (rolling ~6h window, atomic-rename write, every |
| 88 | + failure swallowed). This is **not** a permanent cross-process guarantee: the supervisor's own |
| 89 | + restart limit/backoff (pm2 `max_restarts`, systemd `StartLimitBurst`) remains the outer layer, |
| 90 | + and exit code 75 is only a *request* to respawn. NSSM / Windows services are not auto-detected — |
| 91 | + set `OCX_SUPERVISED=1` there. |
| 92 | + |
| 93 | +## Verification evidence (as merged) |
| 94 | + |
| 95 | +- Unit/integration: `tests/memory-watchdog.test.ts` (66), `tests/memory-restart-history.test.ts` |
| 96 | + (11), `tests/memory-api.test.ts` (16) — includes a REAL Windows end-to-end probe test asserting |
| 97 | + private bytes and a sane commit fraction from the live labeled command. |
| 98 | +- Live measurements on a Windows 11 box: probe cold spawn 4.0–5.9s (hence the 15s async budget); |
| 99 | + `Win32_PerfRawData_PerfOS_Memory` values confirmed to be bytes; harness detectability |
| 100 | + `afterClear private/RSS = 4.4–4.8`. |
| 101 | +- Gates: `typecheck`, `lint:gui`, `build:gui`, `privacy:scan` all exit 0. `probeError` carries |
| 102 | + sanitized codes only (never raw command output/paths) — privacy-scan enforced. |
| 103 | +- Full-suite caveat: on the development Windows box the suite exits 1 due to pre-existing |
| 104 | + environment failures (NTFS ACL hardening `EACCES` clusters in auth tests + branch-dependent |
| 105 | + release-helper tests). Evidence that these are unrelated: the failure set at this branch's HEAD |
| 106 | + is a strict subset (321 ⊂ 345, zero new files, zero memory-file failures) of the same run at the |
| 107 | + pre-change baseline commit under identical conditions. |
| 108 | + |
| 109 | +## Audit trail |
| 110 | + |
| 111 | +Three independent audit rounds shaped this design (summarized honestly rather than re-litigated): |
| 112 | + |
| 113 | +1. Initial audit (deploy blockers): synchronous `spawnSync` probe stall, missing system-commit |
| 114 | + measurement, 60s startup blind spot, RSS-only stress assertions, unwired response-state |
| 115 | + metrics → all five confirmed against code and fixed by the async rewrite. |
| 116 | +2. Design review: chose "cached-free" probe-completion-driven evaluation over defer-until-idle |
| 117 | + and blue-green alternatives (complexity/regression cost vs. benefit), kept warn-only defaults. |
| 118 | +3. Post-implementation audit: auto-restart race in `probeOnce` (unhandled TypeError), capture-throw |
| 119 | + silence, positional PowerShell parsing, live re-clamp gaps → all four fixed with regression |
| 120 | + tests (the race test was validated to fail against the pre-fix code). |
| 121 | + |
| 122 | +## Follow-ups (out of scope here) |
| 123 | + |
| 124 | +- Field measurement, then possible commit-axis restart integration + UI thresholds (v2). |
| 125 | +- Linux system commit via `/proc/meminfo` (`Committed_AS`/`CommitLimit`, no spawn). |
| 126 | +- docs-site ja/zh-cn translations of the memoryWatchdog reference section. |
| 127 | +- Separate issue: `windows-secret-acl` hardening is non-atomic under icacls timeouts |
| 128 | + (inheritance stripped before grant), which is the root cause of the environmental `EACCES` |
| 129 | + test failures on slow-ACL machines. |
0 commit comments