Skip to content

Commit 132f5e6

Browse files
t3dotggclaude
andcommitted
docs(observability): describe the stall window as since the previous sample
A stall delays the next sample, so the gauge and span attributes can cover more than 30 s. Say so in the gauge description and the docs, and explain the perf_hooks diagnostic suppression. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent df2d251 commit 132f5e6

3 files changed

Lines changed: 8 additions & 6 deletions

File tree

‎apps/server/src/observability/EventLoopMonitor.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// @effect-diagnostics nodeBuiltinImport:off
1+
// @effect-diagnostics nodeBuiltinImport:off - only node:perf_hooks exposes the event loop delay histogram.
22
import * as NodePerfHooks from "node:perf_hooks";
33

44
import * as Effect from "effect/Effect";

‎apps/server/src/observability/Metrics.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,8 @@ export const terminalRestartsTotal = Metric.counter("t3_terminal_restarts_total"
7575
});
7676

7777
export const eventLoopDelayMax = Metric.gauge("t3_event_loop_delay_max_ms", {
78-
description: "Longest event loop delay in the last 30 s sample, in milliseconds.",
78+
description:
79+
"Longest event loop delay since the previous sample (nominally 30 s), in milliseconds.",
7980
});
8081

8182
export const metricAttributes = (

‎docs/operations/observability.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -75,12 +75,13 @@ If OTLP is not configured, metrics still exist in-process, but you will not have
7575
### Event Loop Stalls
7676

7777
`apps/server/src/observability/EventLoopMonitor.ts` samples the server's event loop every 30 s. When
78-
the loop was blocked for more than 1 s in that window, it records a root `server.eventLoop.stall`
78+
the loop was blocked for more than 1 s since the previous sample, it records a root `server.eventLoop.stall`
7979
span with a warning. The span has trace level `Warn`, so it stays when `T3CODE_TRACE_MIN_LEVEL` is
8080
`Warn`. The span lands in the trace file, and the warning shows in Settings > Diagnostics unless
81-
OTLP logs are on. The span time is when the sample ran; the stall happened in the 30 s before it.
81+
OTLP logs are on. The span time is when the sample ran; the stall happened since the previous sample.
8282

83-
Attributes cover that 30 s window:
83+
Attributes cover the window since the previous sample. That is nominally 30 s, but a stall delays the
84+
next sample, so a long stall makes the window longer:
8485

8586
- `delayMaxMs`, `delayP99Ms`, `delayMeanMs`: how late the loop ran. The max can undercount a stall
8687
by up to 200 ms.
@@ -90,7 +91,7 @@ Attributes cover that 30 s window:
9091
- `involuntaryContextSwitches`: times the OS took the CPU away from the process.
9192
- `rssMb`: resident memory at sample time.
9293

93-
To read a stall, remember that the CPU times cover the full 30 s window and all threads, but
94+
To read a stall, remember that the CPU times cover the full window and all threads, but
9495
`delayMaxMs` is one stall. Other work in the same window can hide a wait. Only CPU time far below
9596
`delayMaxMs` shows that the thread was waiting. In other cases, read the user and system CPU split
9697
together with the page faults:

0 commit comments

Comments
 (0)