Skip to content

Commit e905bde

Browse files
donjor-agent[bot]rsi-pr-agent
authored andcommitted
fix(server): record main-thread stacks when memory bursts past 4 GiB (#1719)
## Problem In the 2026-10-12 burst, the event loop was blocked from the start. Every timer and log stopped, so the memory watch from #1718 cannot fire in that case. Signal-based reports wait for the loop too. ## Change `startStackWatch` in `MemoryWatch.fork.ts`, started by the existing layer: ### Capture - **Runs on:** a worker thread, off the main loop - **Trigger:** process RSS past 4 GiB - **Action:** inspector `Debugger.pause`, 3 times - **Interval:** 1 s between pauses ### Output - **Output:** `server-<pid>-<time>.stacks.json` - **Location:** `logsDir`, mode 0600 - **Released:** with the layer scope The inspector interrupts running JavaScript, so each pause names the stuck frame. ## Scope and approval Refs #1717. The maintainer asked for the root cause; this captures it on the next burst. No main-thread cost below 4 GiB; the worker only reads RSS once a second. ## Verification - **Unit test:** 2 of 2 pass - **Covers:** a 3 s sync loop on the main thread - **Asserts:** first frame is the stuck function - **Typecheck:** `apps/server`, exit 0 - **Fork CI:** `vp run fork:ci`, exit 0 - **Not checked:** a live 4 GiB burst; no repro yet Claude Code, claude-opus-5-5. ## Fork trailers Squashes: - 0b7add0 fix(server): record main-thread stacks when memory bursts past 4 GiB (#1719) (#1719) - 03a61db fixup! fix(server): record main-thread stacks when memory bursts past 4 GiB (#1719) (#1719) - 75a24fc fix(server): record main-thread stacks when memory bursts past 4 GiB (#1719) (#1719) - d85d60d fixup! fix(server): record main-thread stacks when memory bursts past 4 GiB (#1719) (#1719) Fork-Domain: upstream-fixes Fork-Tier: bugfix Fork-Upstreamable: yes
1 parent afe2734 commit e905bde

2 files changed

Lines changed: 101 additions & 5 deletions

File tree

‎apps/server/src/observability/MemoryWatch.fork.test.ts‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
1+
// @effect-diagnostics nodeBuiltinImport:off - the watched loop must block the real main thread.
2+
import * as NodeFS from "node:fs";
3+
import * as NodeOS from "node:os";
4+
import * as NodePath from "node:path";
5+
16
import { describe, expect, it } from "vite-plus/test";
27

3-
import { initialState, step } from "./MemoryWatch.fork.ts";
8+
import { initialState, startStackWatch, step } from "./MemoryWatch.fork.ts";
49

510
const GiB = 1024 ** 3;
611

@@ -29,3 +34,28 @@ describe("MemoryWatch step", () => {
2934
]);
3035
});
3136
});
37+
38+
describe("startStackWatch", () => {
39+
it("records the frames of a main thread stuck in a synchronous loop", async () => {
40+
const logsDir = NodeFS.mkdtempSync(NodePath.join(NodeOS.tmpdir(), "memory-watch-"));
41+
const worker = startStackWatch({ logsDir, thresholdBytes: 0, samples: 2 });
42+
try {
43+
await new Promise((resolve) => worker.once("online", resolve));
44+
// Spins until the worker writes its stacks; checks are rare so pauses land in the spin,
45+
// and the check bound turns a silent worker into a failure instead of a hung runner.
46+
(function stuckInSyncLoop() {
47+
for (let checks = 0; checks < 1_000 && NodeFS.readdirSync(logsDir).length === 0; checks++) {
48+
for (let spin = 0; spin < 100_000_000; spin++);
49+
}
50+
})();
51+
expect(NodeFS.readdirSync(logsDir)).toHaveLength(1);
52+
const [file] = NodeFS.readdirSync(logsDir);
53+
const stacks = JSON.parse(NodeFS.readFileSync(NodePath.join(logsDir, file!), "utf8"));
54+
expect(stacks).toHaveLength(2);
55+
expect(stacks[0].frames[0]).toMatch(/^stuckInSyncLoop /);
56+
} finally {
57+
await worker.terminate();
58+
NodeFS.rmSync(logsDir, { recursive: true, force: true });
59+
}
60+
});
61+
});

‎apps/server/src/observability/MemoryWatch.fork.ts‎

Lines changed: 70 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
// @effect-diagnostics nodeBuiltinImport:off - the stack watch must run on its own Node thread.
2+
import * as NodeWorkerThreads from "node:worker_threads";
3+
14
import * as Effect from "effect/Effect";
25
import * as Layer from "effect/Layer";
36

@@ -40,15 +43,78 @@ export const step = (state: MemoryWatchState, rssBytes: number) => {
4043
};
4144
};
4245

46+
// Runs on a worker thread, so it still samples when the main thread is stuck
47+
// in a synchronous loop that starves every timer, log, and signal handler.
48+
// The inspector interrupts running JavaScript, so each pause shows the frames
49+
// that are executing right then.
50+
const STACK_WATCH_SOURCE = `
51+
const { workerData } = process.getBuiltinModule("node:worker_threads");
52+
const inspector = process.getBuiltinModule("node:inspector");
53+
const fs = process.getBuiltinModule("node:fs");
54+
const path = process.getBuiltinModule("node:path");
55+
const { logsDir, thresholdBytes, samples } = workerData;
56+
const stacks = [];
57+
let session;
58+
// The interval also keeps this thread alive; inspector callbacks do not.
59+
const timer = setInterval(() => {
60+
if (session === undefined) {
61+
if (process.memoryUsage.rss() < thresholdBytes) return;
62+
session = new inspector.Session();
63+
session.connectToMainThread();
64+
session.on("Debugger.paused", ({ params }) => {
65+
stacks.push({
66+
at: new Date().toISOString(),
67+
rssBytes: process.memoryUsage.rss(),
68+
frames: params.callFrames.map(
69+
(frame) =>
70+
(frame.functionName || "<anonymous>") + " " + frame.url + ":" +
71+
(frame.location.lineNumber + 1) + ":" + (frame.location.columnNumber + 1),
72+
),
73+
});
74+
session.post("Debugger.resume");
75+
if (stacks.length < samples) return;
76+
clearInterval(timer);
77+
const name = "server-" + process.pid + "-" + new Date().toISOString().replaceAll(":", "-") + ".stacks.json";
78+
fs.writeFileSync(path.join(logsDir, name), JSON.stringify(stacks, null, 2), { mode: 0o600 });
79+
session.disconnect();
80+
});
81+
session.post("Debugger.enable");
82+
}
83+
session.post("Debugger.pause");
84+
}, 1000);
85+
`;
86+
87+
/**
88+
* Starts a worker that, once the process passes `thresholdBytes` RSS, pauses
89+
* the main thread `samples` times a second apart and writes the stacks to
90+
* `server-<pid>-<time>.stacks.json` in `logsDir`.
91+
*/
92+
export const startStackWatch = (input: {
93+
readonly logsDir: string;
94+
readonly thresholdBytes: number;
95+
readonly samples: number;
96+
}) => {
97+
const worker = new NodeWorkerThreads.Worker(STACK_WATCH_SOURCE, {
98+
eval: true,
99+
workerData: input,
100+
});
101+
worker.unref();
102+
return worker;
103+
};
104+
43105
/**
44-
* Logs the server's memory breakdown at each new high-water mark, so a leak
45-
* leaves a durable record of which pool grew and when. `external` and
46-
* `arrayBuffers` grow outside the V8 heap limit; a heap snapshot names what
47-
* retains them. See RSI-Software/t3code-hyprws#1717.
106+
* Logs the server's memory breakdown at each new high-water mark, so a leak leaves a durable
107+
* record of which pool grew. `external` and `arrayBuffers` grow outside the V8 heap limit; a
108+
* heap snapshot names what retains them. A burst from a blocked event loop never reaches those,
109+
* so a worker records the main thread's stacks instead.
48110
*/
49111
export const layer = Layer.effectDiscard(
50112
Effect.gen(function* () {
51113
const { logsDir } = yield* ServerConfig.ServerConfig;
114+
yield* Effect.acquireRelease(
115+
Effect.sync(() => startStackWatch({ logsDir, thresholdBytes: SNAPSHOT_BYTES, samples: 3 })),
116+
(worker) => Effect.promise(() => worker.terminate()),
117+
);
52118
let state = initialState;
53119
const tick = Effect.gen(function* () {
54120
const usage = process.memoryUsage();

0 commit comments

Comments
 (0)