Skip to content

Commit 0957fd8

Browse files
sahrizviclaude
andcommitted
fix(tracing): address consensus review of PR #867
Addresses the consensus code review of #867 (6 reviewers, 4 of 5 successful externals + Claude flagged the same critical bug at MAJOR/CRITICAL). CRITICAL fix — per-session crash tracking on FileExporter The original PR put `_crashed` as instance state on FileExporter. In the TUI worker (worker.ts:58-74,96-97), a single FileExporter is cached and shared across every session — the spread `[...tracingExporters]` copies the array, not the inner objects, and `Trace.withExporters` only allocates a fresh FileExporter when `options.maxFiles != null` (rarely set in user config). One session's flushSync would permanently mark the shared exporter crashed, silently suppressing endTrace exports for every subsequent session in the worker — a worse regression than the bug the original PR was fixing. Refactor: `private crashedSessions = new Set<string>()` on FileExporter, with `markCrashed(sessionId: string)`. Export checks `crashedSessions.has(trace.sessionId)` at all three checkpoints. The guard now scopes to the actual crashing trace, not the shared instance. Regression test in tracing-rename-race.test.ts: shared FileExporter, Trace A flushSync, Trace B endTrace — asserts B's file lands with status:"completed". Would have caught the original regression. MAJOR fix — endTrace return value after flushSync `endTrace()` short-circuits at entry when `this.crashed` is set and returns `getTracePath()` instead of letting export() bail and returning undefined. Restores the API contract that endTrace returns a usable file path whenever the canonical file exists. Adopts the "crash wins everywhere" policy — once flushSync has fired, the crashed write is authoritative across all exporters; no further exports run. MAJOR fix — TraceExporter interface adds optional markCrashed Replaces the `instanceof FileExporter` dispatch in flushSync with a polymorphic `exp.markCrashed?.(this.sessionId)`. Adds optional `markCrashed?(sessionId: string): void` to the TraceExporter interface. Custom exporters with canonical-file semantics can now participate in the crash guard without modifying Trace. Drops the brittle instanceof check that broke if FileExporter was ever subclassed or duplicated across module boundaries. Minor fixes (PR-introduced issues only) - flush() now drains M2 follow-up snapshots (bounded loop, 16 iters max) so callers see disk == memory after it returns. The previous version only awaited the current promise; a queued follow-up would land after flush() resolved. - Microtask-ordering invariant documented inline at endTrace entry — explains why setting endTraceStarted before the await is airtight. - Test afterEach safety guard restores fs.rename even if a prior test panicked mid-patch, preventing cross-file leakage of the patched fn. - Misleading comment in M2 deterministic test (claimed "no follow-up scheduled") updated to reflect post-fix behavior. - FileExporter naming: `_crashed` → `crashedSessions` (drops the inconsistent underscore prefix; the bare-name style matches Trace.crashed). - markCrashed docstring clarified: "bail at the next checkpoint" rather than overclaiming retroactive cancellation. Tests: - 366 pass (added shared-FileExporter regression test); 0 fail in the tracing suite. Full opencode suite has only pre-existing unrelated failures (tool.registry, sql_analyze, E2E validators). Pre-existing issue noted but NOT fixed in this commit: - `Trace.withExporters` mutates the caller's array (tracing.ts:413-414). GLM-5.1 surfaced this during review. It's pre-existing code (not introduced by this PR series) and unrelated to the M2/M3 fixes. Filing as a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 20bfcd9 commit 0957fd8

3 files changed

Lines changed: 132 additions & 32 deletions

File tree

‎packages/opencode/src/altimate/observability/tracing.ts‎

Lines changed: 75 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,15 @@ export interface TraceFile {
148148
export interface TraceExporter {
149149
readonly name: string
150150
export(trace: TraceFile): Promise<string | undefined>
151+
// altimate_change start — M3 fix: optional crash-guard hook.
152+
// If implemented, Trace.flushSync() calls this with the crashing session's
153+
// id BEFORE writing the canonical crashed file. Exporters that write to a
154+
// shared canonical location (e.g. FileExporter on the local filesystem)
155+
// should use it to suppress in-flight or subsequent exports that would
156+
// race the synchronous crash write. Exporters that POST to remote
157+
// endpoints (e.g. HttpExporter) don't need to implement it.
158+
markCrashed?(sessionId: string): void
159+
// altimate_change end
151160
}
152161

153162
// ---------------------------------------------------------------------------
@@ -165,15 +174,22 @@ export class FileExporter implements TraceExporter {
165174
readonly name = "file"
166175
private dir: string
167176
private maxFiles: number
168-
// altimate_change start — M3 fix: crash-guard for in-flight export() so
169-
// flushSync's synchronous canonical-file write isn't clobbered by a
170-
// late rename. Mirrors the existing `crashed` guard inside Trace.snapshot().
171-
private _crashed = false
172-
/** Mark this exporter as superseded by a synchronous crash write.
173-
* Subsequent or in-flight export() calls bail before they can overwrite
174-
* the canonical file. Idempotent. */
175-
markCrashed() {
176-
this._crashed = true
177+
// altimate_change start — M3 fix: per-session crash-guard.
178+
// We track which sessions have been crashed-out via flushSync rather than
179+
// a single boolean, because the same FileExporter instance is shared
180+
// across many Trace instances in the TUI worker (worker.ts caches a
181+
// single exporter array at module init, then every session does
182+
// `Trace.withExporters([...tracingExporters], ...)` which spreads the
183+
// array but shares the inner objects). A single boolean would cause one
184+
// crashed session to permanently suppress export() for all subsequent
185+
// sessions in the same worker. Keying by sessionId scopes the guard to
186+
// the actual crashing trace.
187+
private crashedSessions = new Set<string>()
188+
/** Mark a session's exports as superseded by a synchronous crash write.
189+
* Subsequent or in-flight export() calls for that session bail at the
190+
* next checkpoint (entry, pre-writeFile, or pre-rename). Idempotent. */
191+
markCrashed(sessionId: string) {
192+
if (sessionId) this.crashedSessions.add(sessionId)
177193
}
178194
// altimate_change end
179195

@@ -183,8 +199,9 @@ export class FileExporter implements TraceExporter {
183199
}
184200

185201
async export(trace: TraceFile): Promise<string | undefined> {
186-
// altimate_change start — M3 fix: bail at entry if already superseded
187-
if (this._crashed) return undefined
202+
// altimate_change start — M3 fix: bail at entry if this session has been
203+
// marked crashed by a concurrent flushSync.
204+
if (this.crashedSessions.has(trace.sessionId)) return undefined
188205
// altimate_change end
189206
let tmpPath: string | undefined
190207
try {
@@ -198,13 +215,13 @@ export class FileExporter implements TraceExporter {
198215
// altimate_change start — M3 fix: re-check before the long writeFile.
199216
// For large traces the writeFile takes 100+ms — a wide window for
200217
// flushSync to interleave.
201-
if (this._crashed) return undefined
218+
if (this.crashedSessions.has(trace.sessionId)) return undefined
202219
// altimate_change end
203220
await fs.writeFile(tmpPath, JSON.stringify(trace, null, 2))
204221
// altimate_change start — M3 fix: re-check before the rename. If
205222
// flushSync ran during the writeFile, drop the tmp instead of
206223
// overwriting flushSync's crashed canonical file.
207-
if (this._crashed) {
224+
if (this.crashedSessions.has(trace.sessionId)) {
208225
await fs.unlink(tmpPath).catch(() => {})
209226
return undefined
210227
}
@@ -858,13 +875,26 @@ export class Trace {
858875
}
859876

860877
/**
861-
* Wait for any pending async snapshot to complete.
862-
* Use from tests that need to read the trace file deterministically after
863-
* a span completion (instead of `await sleep(50)` which races on slow CI runners).
864-
* No-op if no snapshot is in flight.
878+
* Wait for the on-disk snapshot to catch up with in-memory state.
879+
*
880+
* Drains both the current in-flight snapshot and any M2 follow-up snapshot
881+
* scheduled by its `.finally`. Bounded by `MAX_DRAIN_ITER` to guard against
882+
* pathological cases where events arrive faster than they can be drained.
865883
*/
866884
async flush(): Promise<void> {
867-
if (this.snapshotPromise) await this.snapshotPromise.catch(() => {})
885+
// altimate_change start — M2 fix companion: drain follow-up snapshots so
886+
// callers (and tests) see disk == memory after flush() returns. Without
887+
// this drain loop, the M2 fix's queueMicrotask follow-up runs AFTER
888+
// flush() resolves and the disk still trails by one snapshot.
889+
const MAX_DRAIN_ITER = 16
890+
for (let i = 0; i < MAX_DRAIN_ITER; i++) {
891+
if (this.snapshotPromise) await this.snapshotPromise.catch(() => {})
892+
if (!this.snapshotRequestedDuringPending && !this.snapshotPromise) return
893+
// Yield so any queued microtask (the M2 follow-up) gets to run and
894+
// populate snapshotPromise for the next iteration to await.
895+
await new Promise<void>((r) => queueMicrotask(r))
896+
}
897+
// altimate_change end
868898
}
869899

870900
/**
@@ -939,9 +969,24 @@ export class Trace {
939969
// altimate_change start — M2 fix companion: claim the canonical write
940970
// BEFORE awaiting the in-flight snapshot so any follow-up snapshot the
941971
// in-flight one might schedule in its `.finally` is suppressed.
942-
// Without this, the follow-up's queued microtask runs after the await
943-
// but before endTrace's root-span mutation, capturing stale spans.
972+
//
973+
// Microtask-ordering invariant we rely on below: when an in-flight
974+
// snapshot resolves, its `.finally` runs synchronously inside the
975+
// promise chain — and only THEN does the `await this.snapshotPromise`
976+
// (line below) hand control back to this function. By that point, if
977+
// the .finally tried to schedule a follow-up snapshot via
978+
// `queueMicrotask`, the follow-up is queued but hasn't run yet. We set
979+
// `endTraceStarted=true` BEFORE the await, so whether the .finally
980+
// already checked the flag (during the await) or the queued microtask
981+
// runs after this point (calling `snapshot()` which re-checks at line
982+
// ~803), both paths see the flag and bail.
944983
this.endTraceStarted = true
984+
// M3 fix: if flushSync already wrote the canonical crashed file, return
985+
// the path to that file directly. Calling exporters would either bail
986+
// (FileExporter, via the per-session crash guard) or POST stale
987+
// "completed" data (HttpExporter). The crash signal is authoritative
988+
// across exporters once flushSync has fired.
989+
if (this.crashed) return this.getTracePath()
945990
// altimate_change end
946991

947992
// Wait for any in-flight snapshot to complete before final write
@@ -1053,14 +1098,19 @@ export class Trace {
10531098
// Set crashed BEFORE writing so any in-flight async snapshot() will
10541099
// bail out at its rename step instead of clobbering our crashed file.
10551100
this.crashed = true
1056-
// altimate_change start — M3 fix: also notify every FileExporter so any
1057-
// in-flight or future endTrace export() bails before its writeFile or
1058-
// rename can overwrite the canonical file we're about to write below.
1101+
// altimate_change start — M3 fix: notify every exporter that the current
1102+
// session is crashed BEFORE writing the canonical file synchronously
1103+
// below. Exporters that share a canonical write target (e.g. FileExporter)
1104+
// can suppress in-flight or future writes for this session. Polymorphic
1105+
// dispatch via the optional `markCrashed?` method — no instanceof check,
1106+
// so custom exporters with similar semantics participate automatically.
10591107
// Without this, an in-flight `await endTrace()` whose export() is
10601108
// mid-writeFile (a 100+ms window on multi-MB traces) lands its rename
10611109
// after this synchronous write and clobbers the crashed content.
1062-
for (const exp of this.exporters) {
1063-
if (exp instanceof FileExporter) exp.markCrashed()
1110+
if (this.sessionId) {
1111+
for (const exp of this.exporters) {
1112+
exp.markCrashed?.(this.sessionId)
1113+
}
10641114
}
10651115
// altimate_change end
10661116
this.currentGenerationSpanId = undefined

‎packages/opencode/test/altimate/tracing-display-crash.test.ts‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -505,11 +505,11 @@ describe("flushSync — multiple calls", () => {
505505
tracer.flushSync("early crash")
506506
const flushedPath = tracer.getTracePath()!
507507

508-
// After flushSync, endTrace's export() must NOT overwrite the canonical
509-
// crashed file (the FileExporter is now in _crashed=true state, see M3
510-
// fix in tracing.ts). endTrace's export returns undefined so the file
511-
// path comes from the still-valid flushSync write.
512-
await tracer.endTrace()
508+
// After flushSync, endTrace short-circuits and returns the canonical
509+
// crashed file path. The crashed write is authoritative — endTrace
510+
// does not call any exporter once Trace.crashed is set.
511+
const endPath = await tracer.endTrace()
512+
expect(endPath).toBe(flushedPath)
513513

514514
const traceFile: TraceFile = JSON.parse(await fs.readFile(flushedPath, "utf-8"))
515515
expect(traceFile.summary.status).toBe("crashed")

‎packages/opencode/test/altimate/tracing-rename-race.test.ts‎

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,12 @@ describe("trace corruption — flushSync vs in-flight rename race", () => {
3434
})
3535

3636
afterEach(async () => {
37-
;(fs as any).rename = originalRename
37+
// Safety guard: restore fs.rename even if a test failed mid-patch.
38+
// Without this, a patched fs.rename leaks into subsequent test files
39+
// because `fs/promises` is a process-global namespace object.
40+
if ((fs as any).rename !== originalRename) {
41+
;(fs as any).rename = originalRename
42+
}
3843
await fs.rm(tmpDir, { recursive: true, force: true }).catch(() => {})
3944
})
4045

@@ -153,7 +158,9 @@ describe("trace corruption — flushSync vs in-flight rename race", () => {
153158
})
154159

155160
// Within the rename's 150ms delay, fire B, C, D. snapshotPending is true,
156-
// so each snapshot() returns early. No follow-up snapshot is scheduled.
161+
// so each snapshot() returns early — but the M2 fix sets
162+
// snapshotRequestedDuringPending so a follow-up snapshot is scheduled
163+
// (via queueMicrotask in the in-flight snapshot's .finally).
157164
await new Promise((r) => setTimeout(r, 30))
158165
tracer.logToolCall({
159166
tool: "tool-B",
@@ -387,6 +394,49 @@ describe("trace corruption — flushSync vs in-flight rename race", () => {
387394
expect(observedClobber).toBe(0)
388395
}, 120_000)
389396

397+
// Regression for the critical issue raised in the consensus review of PR #867:
398+
// FileExporter._crashed was originally an instance-level boolean, but the TUI
399+
// worker caches a single FileExporter across sessions (worker.ts shares
400+
// `tracingExporters` via `Trace.withExporters([...tracingExporters])`).
401+
// A single boolean meant one crashed session would permanently suppress
402+
// exports for every subsequent session in the same worker. The fix is
403+
// per-session crash tracking via Set<sessionId>.
404+
test("shared FileExporter — flushSync on trace A must NOT suppress trace B's export", async () => {
405+
const sharedExporter = new FileExporter(tmpDir)
406+
407+
// Trace A: flushSync → marks session-A crashed, writes canonical file
408+
const traceA = Trace.withExporters([sharedExporter])
409+
traceA.startTrace("session-A", { prompt: "A" })
410+
await new Promise((r) => setTimeout(r, 50))
411+
traceA.logToolCall({
412+
tool: "tool-a",
413+
callID: "ca",
414+
state: { status: "completed", input: {}, output: "ok", time: { start: 1, end: 2 } },
415+
})
416+
traceA.flushSync("crash-A")
417+
418+
// Trace B: normal lifecycle on the SAME exporter instance — must export cleanly
419+
const traceB = Trace.withExporters([sharedExporter])
420+
traceB.startTrace("session-B", { prompt: "B" })
421+
traceB.logToolCall({
422+
tool: "tool-b",
423+
callID: "cb",
424+
state: { status: "completed", input: {}, output: "ok", time: { start: 1, end: 2 } },
425+
})
426+
const pathB = await traceB.endTrace()
427+
428+
// Trace A's file: crashed (flushSync wins)
429+
const traceFileA: TraceFile = JSON.parse(
430+
await fs.readFile(path.join(tmpDir, "session-A.json"), "utf-8"),
431+
)
432+
expect(traceFileA.summary.status).toBe("crashed")
433+
434+
// Trace B's file: completed (per-session guard scopes A's crash to A only)
435+
expect(pathB).toBe(path.join(tmpDir, "session-B.json"))
436+
const traceFileB: TraceFile = JSON.parse(await fs.readFile(pathB!, "utf-8"))
437+
expect(traceFileB.summary.status).toBe("completed")
438+
})
439+
390440
test("baseline — flushSync alone writes crashed content correctly (no race)", async () => {
391441
// Sanity check: without the rename delay, flushSync's content lands and survives.
392442
// If this fails we have a different bug than the race we're investigating.

0 commit comments

Comments
 (0)