Skip to content

Commit e12cac3

Browse files
sahrizviclaude
andauthored
fix(tracing): close trace corruption in long-running sessions (#865) (#867)
* fix(tracing): close M2 (debounce drop) and M3 (export-vs-flushSync) trace corruption Two confirmed concurrency bugs in `Trace.snapshot()` and `FileExporter.export()` that caused trace files in long-running sessions to either lose events (M2) or show a stale terminating status instead of "crashed" (M3). See umbrella issue #865 for the full reproduction matrix. M2 — debounce drops events with no follow-up snapshot `snapshot()` returns early when `snapshotPending=true` and was never re-scheduled after the in-flight write completed. In bursty turns (an LLM step that fires 5-10 tool calls back-to-back), 7 of 8 events would be dropped from disk until a fresh event arrived in a future turn. If the process exited in the gap, the burst's tail was lost permanently. Natural reproducer fired 20/20 trials. Fix: track `snapshotRequestedDuringPending` whenever `snapshot()` short- circuits. In `.finally()`, if the flag is set and the trace is neither crashed nor ending, schedule exactly one follow-up snapshot via `queueMicrotask`. Bounded: at most one extra write per "burst → quiet" cycle regardless of burst size. M3 — FileExporter.export() races flushSync with no crashed guard `endTrace()` calls `exporter.export()`, which does writeFile(tmp) → rename(tmp, final). The writeFile of a multi-MB trace takes 100+ms, a wide window for `flushSync` to interleave. The `crashed` flag added in 3846387 only guarded `snapshot()`, not `export()`, so flushSync's synchronous crashed write was overwritten by the export's rename. Natural reproducer fired 10/10 trials with ~52 MB traces. Fix: give `FileExporter` a private `_crashed` flag with `markCrashed()`. Check it at entry, before the writeFile, and before the rename (drop tmp and bail in the last case). `flushSync` iterates exporters and calls `markCrashed()` on each `FileExporter` BEFORE its own synchronous write. M2 companion — endTraceStarted gate Once `endTrace()` claims the canonical write, no concurrent snapshot may run. Without this, the M2 follow-up's queued microtask runs after the `await snapshotPromise` in endTrace but BEFORE the root-span endTime mutation, capturing pre-end state and racing to be last writer. Gate the follow-up against an `endTraceStarted` flag set at endTrace entry. M1 (architectural residual) — NOT closed in this PR The TOCTOU between the synchronous `if (this.crashed)` check and the asynchronous kernel `fs.rename` syscall remains. The existing `crashed` flag from 3846387 protects against it on local SSD (microsecond rename window). It can theoretically fire on slow/network filesystems. Documented as a known design hazard in the umbrella issue. Reproducer kept as `test.skip` for local experimentation. Tests: - `test/altimate/tracing-rename-race.test.ts` — reproducers/regressions for all three mechanisms. M2-natural was 20/20 → now 0/20; M3-natural was 10/10 → now 0/10. M1-natural verifies existing protection. - `test/altimate/tracing-display-crash.test.ts:500` updated: `flushSync then endTrace` previously asserted endTrace overwrote flushSync's crashed status. Under the new policy that assertion is inverted — flushSync wins. Closes #865 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(tracing): address consensus review of PR #867 Addresses the consensus code review of #867 (6 reviewers, 4 of 5 successful externals + Claude flagged the same critical bug at MAJOR/CRITICAL). CRITICAL fix — per-session crash tracking on FileExporter The original PR put `_crashed` as instance state on FileExporter. In the TUI worker (worker.ts:58-74,96-97), a single FileExporter is cached and shared across every session — the spread `[...tracingExporters]` copies the array, not the inner objects, and `Trace.withExporters` only allocates a fresh FileExporter when `options.maxFiles != null` (rarely set in user config). One session's flushSync would permanently mark the shared exporter crashed, silently suppressing endTrace exports for every subsequent session in the worker — a worse regression than the bug the original PR was fixing. Refactor: `private crashedSessions = new Set<string>()` on FileExporter, with `markCrashed(sessionId: string)`. Export checks `crashedSessions.has(trace.sessionId)` at all three checkpoints. The guard now scopes to the actual crashing trace, not the shared instance. Regression test in tracing-rename-race.test.ts: shared FileExporter, Trace A flushSync, Trace B endTrace — asserts B's file lands with status:"completed". Would have caught the original regression. MAJOR fix — endTrace return value after flushSync `endTrace()` short-circuits at entry when `this.crashed` is set and returns `getTracePath()` instead of letting export() bail and returning undefined. Restores the API contract that endTrace returns a usable file path whenever the canonical file exists. Adopts the "crash wins everywhere" policy — once flushSync has fired, the crashed write is authoritative across all exporters; no further exports run. MAJOR fix — TraceExporter interface adds optional markCrashed Replaces the `instanceof FileExporter` dispatch in flushSync with a polymorphic `exp.markCrashed?.(this.sessionId)`. Adds optional `markCrashed?(sessionId: string): void` to the TraceExporter interface. Custom exporters with canonical-file semantics can now participate in the crash guard without modifying Trace. Drops the brittle instanceof check that broke if FileExporter was ever subclassed or duplicated across module boundaries. Minor fixes (PR-introduced issues only) - flush() now drains M2 follow-up snapshots (bounded loop, 16 iters max) so callers see disk == memory after it returns. The previous version only awaited the current promise; a queued follow-up would land after flush() resolved. - Microtask-ordering invariant documented inline at endTrace entry — explains why setting endTraceStarted before the await is airtight. - Test afterEach safety guard restores fs.rename even if a prior test panicked mid-patch, preventing cross-file leakage of the patched fn. - Misleading comment in M2 deterministic test (claimed "no follow-up scheduled") updated to reflect post-fix behavior. - FileExporter naming: `_crashed` → `crashedSessions` (drops the inconsistent underscore prefix; the bare-name style matches Trace.crashed). - markCrashed docstring clarified: "bail at the next checkpoint" rather than overclaiming retroactive cancellation. Tests: - 366 pass (added shared-FileExporter regression test); 0 fail in the tracing suite. Full opencode suite has only pre-existing unrelated failures (tool.registry, sql_analyze, E2E validators). Pre-existing issue noted but NOT fixed in this commit: - `Trace.withExporters` mutates the caller's array (tracing.ts:413-414). GLM-5.1 surfaced this during review. It's pre-existing code (not introduced by this PR series) and unrelated to the M2/M3 fixes. Filing as a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(tracing): sanitize sessionId in markCrashed to match export() lookup key (cubic P1) cubic-dev-ai flagged a P1 (confidence 8) in the consensus-review fix: the crash guard could be silently bypassed for any sessionId containing the file-name-unsafe chars `/`, `\`, `.`, or `:`. Six independent LLM reviewers missed this — see memory note on AI-review blind spots for data-flow bugs. The mismatch: - flushSync calls `markCrashed(this.sessionId)` with the RAW value - FileExporter.export checks `crashedSessions.has(trace.sessionId)` where `trace.sessionId` comes from `buildTraceFile` and is sanitized via `replace(/[/\\.:]/g, "_")` For any sessionId containing one of those chars, the Set entry and the lookup key differ, so the guard returns false and export proceeds — the exact scenario the M3 fix was meant to prevent. Fix: sanitize in markCrashed using the same regex, so the Set always stores the canonical (sanitized) key. Lookup-and-store now agree. Regression test added: a sessionId with `:`, `/`, and `.` is marked crashed, then export() is called with the corresponding sanitized trace. Pre-fix: export proceeds and overwrites flushSync's crashed file. Post-fix: export returns undefined; flushSync's file stands. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(test): correct S27 sql_analyze assertion (CI unblock for #867) The test asserted `schema_inspect` IS in the output, contradicting its own name. The production code at sql-analyze.ts:56-58 flips `isRealFailure` to true when the dispatcher result includes an `error` field, which suppresses the progressive-disclosure suggestion. Test name described correct behaviour; assertion did not. Flipping to `not.toContain` matches both the implementation and the test name. Pre-existing fail on main (verified on v0.8.0 release CI run 26734399443) — unrelated to the trace corruption fix but blocking #867's CI. Landing the fix in this PR for the unblock. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Haider <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 6ad8b47 commit e12cac3

4 files changed

Lines changed: 667 additions & 14 deletions

File tree

‎packages/opencode/src/altimate/observability/tracing.ts‎

Lines changed: 145 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,15 @@ export interface TraceFile {
148148
export interface TraceExporter {
149149
readonly name: string
150150
export(trace: TraceFile): Promise<string | undefined>
151+
// altimate_change start — M3 fix: optional crash-guard hook.
152+
// If implemented, Trace.flushSync() calls this with the crashing session's
153+
// id BEFORE writing the canonical crashed file. Exporters that write to a
154+
// shared canonical location (e.g. FileExporter on the local filesystem)
155+
// should use it to suppress in-flight or subsequent exports that would
156+
// race the synchronous crash write. Exporters that POST to remote
157+
// endpoints (e.g. HttpExporter) don't need to implement it.
158+
markCrashed?(sessionId: string): void
159+
// altimate_change end
151160
}
152161

153162
// ---------------------------------------------------------------------------
@@ -165,13 +174,41 @@ export class FileExporter implements TraceExporter {
165174
readonly name = "file"
166175
private dir: string
167176
private maxFiles: number
177+
// altimate_change start — M3 fix: per-session crash-guard.
178+
// We track which sessions have been crashed-out via flushSync rather than
179+
// a single boolean, because the same FileExporter instance is shared
180+
// across many Trace instances in the TUI worker (worker.ts caches a
181+
// single exporter array at module init, then every session does
182+
// `Trace.withExporters([...tracingExporters], ...)` which spreads the
183+
// array but shares the inner objects). A single boolean would cause one
184+
// crashed session to permanently suppress export() for all subsequent
185+
// sessions in the same worker. Keying by sessionId scopes the guard to
186+
// the actual crashing trace.
187+
private crashedSessions = new Set<string>()
188+
/** Mark a session's exports as superseded by a synchronous crash write.
189+
* Subsequent or in-flight export() calls for that session bail at the
190+
* next checkpoint (entry, pre-writeFile, or pre-rename). Idempotent.
191+
* The sessionId is normalized through the same sanitization that
192+
* buildTraceFile / export use for the on-disk file name, so callers
193+
* can pass the raw `this.sessionId` without worrying about whether
194+
* it contains path-unsafe characters. */
195+
markCrashed(sessionId: string) {
196+
if (!sessionId) return
197+
const safeId = sessionId.replace(/[/\\.:]/g, "_") || "unknown"
198+
this.crashedSessions.add(safeId)
199+
}
200+
// altimate_change end
168201

169202
constructor(dir?: string, maxFiles?: number) {
170203
this.dir = dir ?? DEFAULT_TRACES_DIR
171204
this.maxFiles = maxFiles ?? DEFAULT_MAX_FILES
172205
}
173206

174207
async export(trace: TraceFile): Promise<string | undefined> {
208+
// altimate_change start — M3 fix: bail at entry if this session has been
209+
// marked crashed by a concurrent flushSync.
210+
if (this.crashedSessions.has(trace.sessionId)) return undefined
211+
// altimate_change end
175212
let tmpPath: string | undefined
176213
try {
177214
await fs.mkdir(this.dir, { recursive: true })
@@ -181,7 +218,20 @@ export class FileExporter implements TraceExporter {
181218
// Atomic write: write to temp file, then rename — prevents partial reads
182219
// when concurrent snapshots or exports target the same file
183220
tmpPath = filePath + `.tmp.${Date.now()}.${Math.random().toString(36).slice(2, 8)}`
221+
// altimate_change start — M3 fix: re-check before the long writeFile.
222+
// For large traces the writeFile takes 100+ms — a wide window for
223+
// flushSync to interleave.
224+
if (this.crashedSessions.has(trace.sessionId)) return undefined
225+
// altimate_change end
184226
await fs.writeFile(tmpPath, JSON.stringify(trace, null, 2))
227+
// altimate_change start — M3 fix: re-check before the rename. If
228+
// flushSync ran during the writeFile, drop the tmp instead of
229+
// overwriting flushSync's crashed canonical file.
230+
if (this.crashedSessions.has(trace.sessionId)) {
231+
await fs.unlink(tmpPath).catch(() => {})
232+
return undefined
233+
}
234+
// altimate_change end
185235
await fs.rename(tmpPath, filePath)
186236

187237
if (this.maxFiles > 0) {
@@ -337,6 +387,17 @@ export class Trace {
337387
private metadata: TraceFile["metadata"] = {}
338388
private snapshotDir: string | undefined
339389
private snapshotPending = false
390+
// altimate_change start — M2 fix: if a snapshot was requested while another
391+
// was in flight (and thus dropped by the snapshotPending debounce), we
392+
// schedule exactly one follow-up snapshot when the in-flight one finishes.
393+
// Without this, the on-disk file lags memory until a fresh event arrives
394+
// — and if the process exits in that gap, the burst's tail is lost.
395+
private snapshotRequestedDuringPending = false
396+
// Once endTrace begins its canonical write, no more snapshots may run —
397+
// they would race endTrace's mutation of the root span (endTime, status)
398+
// and could clobber endTrace's content with stale pre-end state.
399+
private endTraceStarted = false
400+
// altimate_change end
340401
private snapshotPromise: Promise<void> | undefined
341402
// Set true when flushSync runs. Prevents in-flight async snapshot()
342403
// calls from racing with the synchronous crash write — without this,
@@ -750,9 +811,25 @@ export class Trace {
750811
*/
751812
private snapshot() {
752813
if (!this.snapshotDir || !this.sessionId) return
753-
if (this.snapshotPending) return // Debounce — only one in flight at a time
814+
// altimate_change start — M2 fix: when debounce drops this call, record
815+
// that another snapshot is needed once the in-flight one settles.
816+
// Without this, the disk file lags memory whenever events arrive faster
817+
// than snapshots can complete (typical for bursty LLM turns).
818+
if (this.snapshotPending) {
819+
this.snapshotRequestedDuringPending = true
820+
return
821+
}
822+
// No new snapshots once endTrace has begun: the canonical write is now
823+
// endTrace's job, and a concurrent snapshot would capture state from
824+
// BEFORE endTrace's root-span mutation and could lose to-end-only fields.
825+
if (this.endTraceStarted) return
826+
// altimate_change end
754827
if (this.crashed) return // flushSync wrote the canonical crashed file; do NOT race it
755828
this.snapshotPending = true
829+
// altimate_change start — M2 fix: reset before kicking off, so any
830+
// snapshot() calls during this write set the dirty flag for our .finally.
831+
this.snapshotRequestedDuringPending = false
832+
// altimate_change end
756833

757834
const trace = this.buildTraceFile()
758835
const safeId = (this.sessionId || "unknown").replace(/[/\\.:]/g, "_") || "unknown"
@@ -779,6 +856,17 @@ export class Trace {
779856
.finally(() => {
780857
this.snapshotPending = false
781858
this.snapshotPromise = undefined
859+
// altimate_change start — M2 fix: if any span was added (or any
860+
// snapshot was requested) while we were writing, schedule exactly
861+
// one follow-up to flush those changes. Bounded: at most one extra
862+
// write per "burst → quiet" cycle, regardless of burst size.
863+
// Skip the follow-up if endTrace has taken over the canonical write,
864+
// or if flushSync has marked the trace crashed.
865+
if (this.snapshotRequestedDuringPending && !this.crashed && !this.endTraceStarted) {
866+
this.snapshotRequestedDuringPending = false
867+
queueMicrotask(() => this.snapshot())
868+
}
869+
// altimate_change end
782870
})
783871
}
784872

@@ -793,13 +881,26 @@ export class Trace {
793881
}
794882

795883
/**
796-
* Wait for any pending async snapshot to complete.
797-
* Use from tests that need to read the trace file deterministically after
798-
* a span completion (instead of `await sleep(50)` which races on slow CI runners).
799-
* No-op if no snapshot is in flight.
884+
* Wait for the on-disk snapshot to catch up with in-memory state.
885+
*
886+
* Drains both the current in-flight snapshot and any M2 follow-up snapshot
887+
* scheduled by its `.finally`. Bounded by `MAX_DRAIN_ITER` to guard against
888+
* pathological cases where events arrive faster than they can be drained.
800889
*/
801890
async flush(): Promise<void> {
802-
if (this.snapshotPromise) await this.snapshotPromise.catch(() => {})
891+
// altimate_change start — M2 fix companion: drain follow-up snapshots so
892+
// callers (and tests) see disk == memory after flush() returns. Without
893+
// this drain loop, the M2 fix's queueMicrotask follow-up runs AFTER
894+
// flush() resolves and the disk still trails by one snapshot.
895+
const MAX_DRAIN_ITER = 16
896+
for (let i = 0; i < MAX_DRAIN_ITER; i++) {
897+
if (this.snapshotPromise) await this.snapshotPromise.catch(() => {})
898+
if (!this.snapshotRequestedDuringPending && !this.snapshotPromise) return
899+
// Yield so any queued microtask (the M2 follow-up) gets to run and
900+
// populate snapshotPromise for the next iteration to await.
901+
await new Promise<void>((r) => queueMicrotask(r))
902+
}
903+
// altimate_change end
803904
}
804905

805906
/**
@@ -871,6 +972,29 @@ export class Trace {
871972
* Returns the result from the first exporter that succeeds (typically the file path).
872973
*/
873974
async endTrace(error?: string): Promise<string | undefined> {
975+
// altimate_change start — M2 fix companion: claim the canonical write
976+
// BEFORE awaiting the in-flight snapshot so any follow-up snapshot the
977+
// in-flight one might schedule in its `.finally` is suppressed.
978+
//
979+
// Microtask-ordering invariant we rely on below: when an in-flight
980+
// snapshot resolves, its `.finally` runs synchronously inside the
981+
// promise chain — and only THEN does the `await this.snapshotPromise`
982+
// (line below) hand control back to this function. By that point, if
983+
// the .finally tried to schedule a follow-up snapshot via
984+
// `queueMicrotask`, the follow-up is queued but hasn't run yet. We set
985+
// `endTraceStarted=true` BEFORE the await, so whether the .finally
986+
// already checked the flag (during the await) or the queued microtask
987+
// runs after this point (calling `snapshot()` which re-checks at line
988+
// ~803), both paths see the flag and bail.
989+
this.endTraceStarted = true
990+
// M3 fix: if flushSync already wrote the canonical crashed file, return
991+
// the path to that file directly. Calling exporters would either bail
992+
// (FileExporter, via the per-session crash guard) or POST stale
993+
// "completed" data (HttpExporter). The crash signal is authoritative
994+
// across exporters once flushSync has fired.
995+
if (this.crashed) return this.getTracePath()
996+
// altimate_change end
997+
874998
// Wait for any in-flight snapshot to complete before final write
875999
if (this.snapshotPromise) await this.snapshotPromise.catch(() => {})
8761000

@@ -980,6 +1104,21 @@ export class Trace {
9801104
// Set crashed BEFORE writing so any in-flight async snapshot() will
9811105
// bail out at its rename step instead of clobbering our crashed file.
9821106
this.crashed = true
1107+
// altimate_change start — M3 fix: notify every exporter that the current
1108+
// session is crashed BEFORE writing the canonical file synchronously
1109+
// below. Exporters that share a canonical write target (e.g. FileExporter)
1110+
// can suppress in-flight or future writes for this session. Polymorphic
1111+
// dispatch via the optional `markCrashed?` method — no instanceof check,
1112+
// so custom exporters with similar semantics participate automatically.
1113+
// Without this, an in-flight `await endTrace()` whose export() is
1114+
// mid-writeFile (a 100+ms window on multi-MB traces) lands its rename
1115+
// after this synchronous write and clobbers the crashed content.
1116+
if (this.sessionId) {
1117+
for (const exp of this.exporters) {
1118+
exp.markCrashed?.(this.sessionId)
1119+
}
1120+
}
1121+
// altimate_change end
9831122
this.currentGenerationSpanId = undefined
9841123
const rootSpan = this.spans.find((s) => s.spanId === this.rootSpanId)
9851124
if (rootSpan) {

‎packages/opencode/test/altimate/tracing-display-crash.test.ts‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -497,18 +497,23 @@ describe("flushSync — multiple calls", () => {
497497
expect(traceFile.summary.status).toBe("crashed")
498498
})
499499

500-
test("flushSync then endTrace — endTrace overwrites crashed status", async () => {
500+
test("flushSync then endTrace — flushSync's crashed status is preserved (M3 fix)", async () => {
501501
const tracer = Recap.withExporters([new FileExporter(tmpDir)])
502502
tracer.startTrace("s-flush-then-end", { prompt: "test" })
503503
await new Promise((r) => setTimeout(r, 50))
504504

505505
tracer.flushSync("early crash")
506+
const flushedPath = tracer.getTracePath()!
506507

507-
// But actually the process survived — endTrace completes normally
508-
const filePath = await tracer.endTrace()
509-
const traceFile: TraceFile = JSON.parse(await fs.readFile(filePath!, "utf-8"))
510-
// endTrace should overwrite with "completed"
511-
expect(traceFile.summary.status).toBe("completed")
508+
// After flushSync, endTrace short-circuits and returns the canonical
509+
// crashed file path. The crashed write is authoritative — endTrace
510+
// does not call any exporter once Trace.crashed is set.
511+
const endPath = await tracer.endTrace()
512+
expect(endPath).toBe(flushedPath)
513+
514+
const traceFile: TraceFile = JSON.parse(await fs.readFile(flushedPath, "utf-8"))
515+
expect(traceFile.summary.status).toBe("crashed")
516+
expect(traceFile.summary.error).toContain("early crash")
512517
})
513518
})
514519

0 commit comments

Comments
 (0)