Skip to content

Commit 20bfcd9

Browse files
sahrizviclaude
andcommitted
fix(tracing): close M2 (debounce drop) and M3 (export-vs-flushSync) trace corruption
Two confirmed concurrency bugs in `Trace.snapshot()` and `FileExporter.export()` that caused trace files in long-running sessions to either lose events (M2) or show a stale terminating status instead of "crashed" (M3). See umbrella issue #865 for the full reproduction matrix. M2 — debounce drops events with no follow-up snapshot `snapshot()` returns early when `snapshotPending=true` and was never re-scheduled after the in-flight write completed. In bursty turns (an LLM step that fires 5-10 tool calls back-to-back), 7 of 8 events would be dropped from disk until a fresh event arrived in a future turn. If the process exited in the gap, the burst's tail was lost permanently. Natural reproducer fired 20/20 trials. Fix: track `snapshotRequestedDuringPending` whenever `snapshot()` short- circuits. In `.finally()`, if the flag is set and the trace is neither crashed nor ending, schedule exactly one follow-up snapshot via `queueMicrotask`. Bounded: at most one extra write per "burst → quiet" cycle regardless of burst size. M3 — FileExporter.export() races flushSync with no crashed guard `endTrace()` calls `exporter.export()`, which does writeFile(tmp) → rename(tmp, final). The writeFile of a multi-MB trace takes 100+ms, a wide window for `flushSync` to interleave. The `crashed` flag added in 3846387 only guarded `snapshot()`, not `export()`, so flushSync's synchronous crashed write was overwritten by the export's rename. Natural reproducer fired 10/10 trials with ~52 MB traces. Fix: give `FileExporter` a private `_crashed` flag with `markCrashed()`. Check it at entry, before the writeFile, and before the rename (drop tmp and bail in the last case). `flushSync` iterates exporters and calls `markCrashed()` on each `FileExporter` BEFORE its own synchronous write. M2 companion — endTraceStarted gate Once `endTrace()` claims the canonical write, no concurrent snapshot may run. Without this, the M2 follow-up's queued microtask runs after the `await snapshotPromise` in endTrace but BEFORE the root-span endTime mutation, capturing pre-end state and racing to be last writer. Gate the follow-up against an `endTraceStarted` flag set at endTrace entry. M1 (architectural residual) — NOT closed in this PR The TOCTOU between the synchronous `if (this.crashed)` check and the asynchronous kernel `fs.rename` syscall remains. The existing `crashed` flag from 3846387 protects against it on local SSD (microsecond rename window). It can theoretically fire on slow/network filesystems. Documented as a known design hazard in the umbrella issue. Reproducer kept as `test.skip` for local experimentation. Tests: - `test/altimate/tracing-rename-race.test.ts` — reproducers/regressions for all three mechanisms. M2-natural was 20/20 → now 0/20; M3-natural was 10/10 → now 0/10. M1-natural verifies existing protection. - `test/altimate/tracing-display-crash.test.ts:500` updated: `flushSync then endTrace` previously asserted endTrace overwrote flushSync's crashed status. Under the new policy that assertion is inverted — flushSync wins. Closes #865 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 6ad8b47 commit 20bfcd9

3 files changed

Lines changed: 502 additions & 7 deletions

File tree

‎packages/opencode/src/altimate/observability/tracing.ts‎

Lines changed: 84 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,13 +165,27 @@ export class FileExporter implements TraceExporter {
165165
readonly name = "file"
166166
private dir: string
167167
private maxFiles: number
168+
// altimate_change start — M3 fix: crash-guard for in-flight export() so
169+
// flushSync's synchronous canonical-file write isn't clobbered by a
170+
// late rename. Mirrors the existing `crashed` guard inside Trace.snapshot().
171+
private _crashed = false
172+
/** Mark this exporter as superseded by a synchronous crash write.
173+
* Subsequent or in-flight export() calls bail before they can overwrite
174+
* the canonical file. Idempotent. */
175+
markCrashed() {
176+
this._crashed = true
177+
}
178+
// altimate_change end
168179

169180
constructor(dir?: string, maxFiles?: number) {
170181
this.dir = dir ?? DEFAULT_TRACES_DIR
171182
this.maxFiles = maxFiles ?? DEFAULT_MAX_FILES
172183
}
173184

174185
async export(trace: TraceFile): Promise<string | undefined> {
186+
// altimate_change start — M3 fix: bail at entry if already superseded
187+
if (this._crashed) return undefined
188+
// altimate_change end
175189
let tmpPath: string | undefined
176190
try {
177191
await fs.mkdir(this.dir, { recursive: true })
@@ -181,7 +195,20 @@ export class FileExporter implements TraceExporter {
181195
// Atomic write: write to temp file, then rename — prevents partial reads
182196
// when concurrent snapshots or exports target the same file
183197
tmpPath = filePath + `.tmp.${Date.now()}.${Math.random().toString(36).slice(2, 8)}`
198+
// altimate_change start — M3 fix: re-check before the long writeFile.
199+
// For large traces the writeFile takes 100+ms — a wide window for
200+
// flushSync to interleave.
201+
if (this._crashed) return undefined
202+
// altimate_change end
184203
await fs.writeFile(tmpPath, JSON.stringify(trace, null, 2))
204+
// altimate_change start — M3 fix: re-check before the rename. If
205+
// flushSync ran during the writeFile, drop the tmp instead of
206+
// overwriting flushSync's crashed canonical file.
207+
if (this._crashed) {
208+
await fs.unlink(tmpPath).catch(() => {})
209+
return undefined
210+
}
211+
// altimate_change end
185212
await fs.rename(tmpPath, filePath)
186213

187214
if (this.maxFiles > 0) {
@@ -337,6 +364,17 @@ export class Trace {
337364
private metadata: TraceFile["metadata"] = {}
338365
private snapshotDir: string | undefined
339366
private snapshotPending = false
367+
// altimate_change start — M2 fix: if a snapshot was requested while another
368+
// was in flight (and thus dropped by the snapshotPending debounce), we
369+
// schedule exactly one follow-up snapshot when the in-flight one finishes.
370+
// Without this, the on-disk file lags memory until a fresh event arrives
371+
// — and if the process exits in that gap, the burst's tail is lost.
372+
private snapshotRequestedDuringPending = false
373+
// Once endTrace begins its canonical write, no more snapshots may run —
374+
// they would race endTrace's mutation of the root span (endTime, status)
375+
// and could clobber endTrace's content with stale pre-end state.
376+
private endTraceStarted = false
377+
// altimate_change end
340378
private snapshotPromise: Promise<void> | undefined
341379
// Set true when flushSync runs. Prevents in-flight async snapshot()
342380
// calls from racing with the synchronous crash write — without this,
@@ -750,9 +788,25 @@ export class Trace {
750788
*/
751789
private snapshot() {
752790
if (!this.snapshotDir || !this.sessionId) return
753-
if (this.snapshotPending) return // Debounce — only one in flight at a time
791+
// altimate_change start — M2 fix: when debounce drops this call, record
792+
// that another snapshot is needed once the in-flight one settles.
793+
// Without this, the disk file lags memory whenever events arrive faster
794+
// than snapshots can complete (typical for bursty LLM turns).
795+
if (this.snapshotPending) {
796+
this.snapshotRequestedDuringPending = true
797+
return
798+
}
799+
// No new snapshots once endTrace has begun: the canonical write is now
800+
// endTrace's job, and a concurrent snapshot would capture state from
801+
// BEFORE endTrace's root-span mutation and could lose to-end-only fields.
802+
if (this.endTraceStarted) return
803+
// altimate_change end
754804
if (this.crashed) return // flushSync wrote the canonical crashed file; do NOT race it
755805
this.snapshotPending = true
806+
// altimate_change start — M2 fix: reset before kicking off, so any
807+
// snapshot() calls during this write set the dirty flag for our .finally.
808+
this.snapshotRequestedDuringPending = false
809+
// altimate_change end
756810

757811
const trace = this.buildTraceFile()
758812
const safeId = (this.sessionId || "unknown").replace(/[/\\.:]/g, "_") || "unknown"
@@ -779,6 +833,17 @@ export class Trace {
779833
.finally(() => {
780834
this.snapshotPending = false
781835
this.snapshotPromise = undefined
836+
// altimate_change start — M2 fix: if any span was added (or any
837+
// snapshot was requested) while we were writing, schedule exactly
838+
// one follow-up to flush those changes. Bounded: at most one extra
839+
// write per "burst → quiet" cycle, regardless of burst size.
840+
// Skip the follow-up if endTrace has taken over the canonical write,
841+
// or if flushSync has marked the trace crashed.
842+
if (this.snapshotRequestedDuringPending && !this.crashed && !this.endTraceStarted) {
843+
this.snapshotRequestedDuringPending = false
844+
queueMicrotask(() => this.snapshot())
845+
}
846+
// altimate_change end
782847
})
783848
}
784849

@@ -871,6 +936,14 @@ export class Trace {
871936
* Returns the result from the first exporter that succeeds (typically the file path).
872937
*/
873938
async endTrace(error?: string): Promise<string | undefined> {
939+
// altimate_change start — M2 fix companion: claim the canonical write
940+
// BEFORE awaiting the in-flight snapshot so any follow-up snapshot the
941+
// in-flight one might schedule in its `.finally` is suppressed.
942+
// Without this, the follow-up's queued microtask runs after the await
943+
// but before endTrace's root-span mutation, capturing stale spans.
944+
this.endTraceStarted = true
945+
// altimate_change end
946+
874947
// Wait for any in-flight snapshot to complete before final write
875948
if (this.snapshotPromise) await this.snapshotPromise.catch(() => {})
876949

@@ -980,6 +1053,16 @@ export class Trace {
9801053
// Set crashed BEFORE writing so any in-flight async snapshot() will
9811054
// bail out at its rename step instead of clobbering our crashed file.
9821055
this.crashed = true
1056+
// altimate_change start — M3 fix: also notify every FileExporter so any
1057+
// in-flight or future endTrace export() bails before its writeFile or
1058+
// rename can overwrite the canonical file we're about to write below.
1059+
// Without this, an in-flight `await endTrace()` whose export() is
1060+
// mid-writeFile (a 100+ms window on multi-MB traces) lands its rename
1061+
// after this synchronous write and clobbers the crashed content.
1062+
for (const exp of this.exporters) {
1063+
if (exp instanceof FileExporter) exp.markCrashed()
1064+
}
1065+
// altimate_change end
9831066
this.currentGenerationSpanId = undefined
9841067
const rootSpan = this.spans.find((s) => s.spanId === this.rootSpanId)
9851068
if (rootSpan) {

‎packages/opencode/test/altimate/tracing-display-crash.test.ts‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -497,18 +497,23 @@ describe("flushSync — multiple calls", () => {
497497
expect(traceFile.summary.status).toBe("crashed")
498498
})
499499

500-
test("flushSync then endTrace — endTrace overwrites crashed status", async () => {
500+
test("flushSync then endTrace — flushSync's crashed status is preserved (M3 fix)", async () => {
501501
const tracer = Recap.withExporters([new FileExporter(tmpDir)])
502502
tracer.startTrace("s-flush-then-end", { prompt: "test" })
503503
await new Promise((r) => setTimeout(r, 50))
504504

505505
tracer.flushSync("early crash")
506+
const flushedPath = tracer.getTracePath()!
506507

507-
// But actually the process survived — endTrace completes normally
508-
const filePath = await tracer.endTrace()
509-
const traceFile: TraceFile = JSON.parse(await fs.readFile(filePath!, "utf-8"))
510-
// endTrace should overwrite with "completed"
511-
expect(traceFile.summary.status).toBe("completed")
508+
// After flushSync, endTrace's export() must NOT overwrite the canonical
509+
// crashed file (the FileExporter is now in _crashed=true state, see M3
510+
// fix in tracing.ts). endTrace's export returns undefined so the file
511+
// path comes from the still-valid flushSync write.
512+
await tracer.endTrace()
513+
514+
const traceFile: TraceFile = JSON.parse(await fs.readFile(flushedPath, "utf-8"))
515+
expect(traceFile.summary.status).toBe("crashed")
516+
expect(traceFile.summary.error).toContain("early crash")
512517
})
513518
})
514519

0 commit comments

Comments
 (0)