Repository navigation
Commit e12cac3
* fix(tracing): close M2 (debounce drop) and M3 (export-vs-flushSync) trace corruption
Two confirmed concurrency bugs in `Trace.snapshot()` and `FileExporter.export()`
that caused trace files in long-running sessions to either lose events (M2) or
show a stale terminating status instead of "crashed" (M3). See umbrella issue
#865 for the full reproduction matrix.
M2 — debounce drops events with no follow-up snapshot
`snapshot()` returns early when `snapshotPending=true` and was never
re-scheduled after the in-flight write completed. In bursty turns (an LLM
step that fires 5-10 tool calls back-to-back), 7 of 8 events would be
dropped from disk until a fresh event arrived in a future turn. If the
process exited in the gap, the burst's tail was lost permanently. Natural
reproducer fired 20/20 trials.
Fix: track `snapshotRequestedDuringPending` whenever `snapshot()` short-
circuits. In `.finally()`, if the flag is set and the trace is neither
crashed nor ending, schedule exactly one follow-up snapshot via
`queueMicrotask`. Bounded: at most one extra write per "burst → quiet"
cycle regardless of burst size.
M3 — FileExporter.export() races flushSync with no crashed guard
`endTrace()` calls `exporter.export()`, which does
writeFile(tmp) → rename(tmp, final). The writeFile of a multi-MB trace
takes 100+ms, a wide window for `flushSync` to interleave. The `crashed`
flag added in 3846387 only guarded `snapshot()`, not `export()`, so
flushSync's synchronous crashed write was overwritten by the export's
rename. Natural reproducer fired 10/10 trials with ~52 MB traces.
Fix: give `FileExporter` a private `_crashed` flag with `markCrashed()`.
Check it at entry, before the writeFile, and before the rename (drop tmp
and bail in the last case). `flushSync` iterates exporters and calls
`markCrashed()` on each `FileExporter` BEFORE its own synchronous write.
M2 companion — endTraceStarted gate
Once `endTrace()` claims the canonical write, no concurrent snapshot may
run. Without this, the M2 follow-up's queued microtask runs after the
`await snapshotPromise` in endTrace but BEFORE the root-span endTime
mutation, capturing pre-end state and racing to be last writer. Gate the
follow-up against an `endTraceStarted` flag set at endTrace entry.
M1 (architectural residual) — NOT closed in this PR
The TOCTOU between the synchronous `if (this.crashed)` check and the
asynchronous kernel `fs.rename` syscall remains. The existing `crashed`
flag from 3846387 protects against it on local SSD (microsecond rename
window). It can theoretically fire on slow/network filesystems. Documented
as a known design hazard in the umbrella issue. Reproducer kept as
`test.skip` for local experimentation.
Tests:
- `test/altimate/tracing-rename-race.test.ts` — reproducers/regressions
for all three mechanisms. M2-natural was 20/20 → now 0/20; M3-natural
was 10/10 → now 0/10. M1-natural verifies existing protection.
- `test/altimate/tracing-display-crash.test.ts:500` updated:
`flushSync then endTrace` previously asserted endTrace overwrote
flushSync's crashed status. Under the new policy that assertion is
inverted — flushSync wins.
Closes #865
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(tracing): address consensus review of PR #867
Addresses the consensus code review of #867 (6 reviewers, 4 of 5 successful
externals + Claude flagged the same critical bug at MAJOR/CRITICAL).
CRITICAL fix — per-session crash tracking on FileExporter
The original PR put `_crashed` as instance state on FileExporter. In the
TUI worker (worker.ts:58-74,96-97), a single FileExporter is cached and
shared across every session — the spread `[...tracingExporters]` copies
the array, not the inner objects, and `Trace.withExporters` only allocates
a fresh FileExporter when `options.maxFiles != null` (rarely set in user
config). One session's flushSync would permanently mark the shared
exporter crashed, silently suppressing endTrace exports for every
subsequent session in the worker — a worse regression than the bug the
original PR was fixing.
Refactor: `private crashedSessions = new Set<string>()` on FileExporter,
with `markCrashed(sessionId: string)`. Export checks
`crashedSessions.has(trace.sessionId)` at all three checkpoints. The
guard now scopes to the actual crashing trace, not the shared instance.
Regression test in tracing-rename-race.test.ts: shared FileExporter,
Trace A flushSync, Trace B endTrace — asserts B's file lands with
status:"completed". Would have caught the original regression.
MAJOR fix — endTrace return value after flushSync
`endTrace()` short-circuits at entry when `this.crashed` is set and
returns `getTracePath()` instead of letting export() bail and returning
undefined. Restores the API contract that endTrace returns a usable file
path whenever the canonical file exists. Adopts the "crash wins
everywhere" policy — once flushSync has fired, the crashed write is
authoritative across all exporters; no further exports run.
MAJOR fix — TraceExporter interface adds optional markCrashed
Replaces the `instanceof FileExporter` dispatch in flushSync with a
polymorphic `exp.markCrashed?.(this.sessionId)`. Adds optional
`markCrashed?(sessionId: string): void` to the TraceExporter interface.
Custom exporters with canonical-file semantics can now participate in
the crash guard without modifying Trace. Drops the brittle instanceof
check that broke if FileExporter was ever subclassed or duplicated
across module boundaries.
Minor fixes (PR-introduced issues only)
- flush() now drains M2 follow-up snapshots (bounded loop, 16 iters max)
so callers see disk == memory after it returns. The previous version
only awaited the current promise; a queued follow-up would land after
flush() resolved.
- Microtask-ordering invariant documented inline at endTrace entry —
explains why setting endTraceStarted before the await is airtight.
- Test afterEach safety guard restores fs.rename even if a prior test
panicked mid-patch, preventing cross-file leakage of the patched fn.
- Misleading comment in M2 deterministic test (claimed "no follow-up
scheduled") updated to reflect post-fix behavior.
- FileExporter naming: `_crashed` → `crashedSessions` (drops the
inconsistent underscore prefix; the bare-name style matches Trace.crashed).
- markCrashed docstring clarified: "bail at the next checkpoint" rather
than overclaiming retroactive cancellation.
Tests:
- 366 pass (added shared-FileExporter regression test); 0 fail in the
tracing suite. Full opencode suite has only pre-existing unrelated
failures (tool.registry, sql_analyze, E2E validators).
Pre-existing issue noted but NOT fixed in this commit:
- `Trace.withExporters` mutates the caller's array (tracing.ts:413-414).
GLM-5.1 surfaced this during review. It's pre-existing code (not
introduced by this PR series) and unrelated to the M2/M3 fixes. Filing
as a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(tracing): sanitize sessionId in markCrashed to match export() lookup key (cubic P1)
cubic-dev-ai flagged a P1 (confidence 8) in the consensus-review fix: the
crash guard could be silently bypassed for any sessionId containing the
file-name-unsafe chars `/`, `\`, `.`, or `:`. Six independent LLM reviewers
missed this — see memory note on AI-review blind spots for data-flow bugs.
The mismatch:
- flushSync calls `markCrashed(this.sessionId)` with the RAW value
- FileExporter.export checks `crashedSessions.has(trace.sessionId)`
where `trace.sessionId` comes from `buildTraceFile` and is sanitized
via `replace(/[/\\.:]/g, "_")`
For any sessionId containing one of those chars, the Set entry and the
lookup key differ, so the guard returns false and export proceeds — the
exact scenario the M3 fix was meant to prevent.
Fix: sanitize in markCrashed using the same regex, so the Set always
stores the canonical (sanitized) key. Lookup-and-store now agree.
Regression test added: a sessionId with `:`, `/`, and `.` is marked
crashed, then export() is called with the corresponding sanitized trace.
Pre-fix: export proceeds and overwrites flushSync's crashed file.
Post-fix: export returns undefined; flushSync's file stands.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(test): correct S27 sql_analyze assertion (CI unblock for #867)
The test asserted `schema_inspect` IS in the output, contradicting its
own name. The production code at sql-analyze.ts:56-58 flips
`isRealFailure` to true when the dispatcher result includes an
`error` field, which suppresses the progressive-disclosure suggestion.
Test name described correct behaviour; assertion did not. Flipping to
`not.toContain` matches both the implementation and the test name.
Pre-existing fail on main (verified on v0.8.0 release CI run
26734399443) — unrelated to the trace corruption fix but blocking
#867's CI. Landing the fix in this PR for the unblock.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: Haider <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 6ad8b47 commit e12cac3
4 files changed
Lines changed: 667 additions & 14 deletions
File tree
- packages/opencode
- src/altimate/observability
- test
- altimate
- session
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
148 | 148 | | |
149 | 149 | | |
150 | 150 | | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
151 | 160 | | |
152 | 161 | | |
153 | 162 | | |
| |||
165 | 174 | | |
166 | 175 | | |
167 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
168 | 201 | | |
169 | 202 | | |
170 | 203 | | |
171 | 204 | | |
172 | 205 | | |
173 | 206 | | |
174 | 207 | | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
175 | 212 | | |
176 | 213 | | |
177 | 214 | | |
| |||
181 | 218 | | |
182 | 219 | | |
183 | 220 | | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
184 | 226 | | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
185 | 235 | | |
186 | 236 | | |
187 | 237 | | |
| |||
337 | 387 | | |
338 | 388 | | |
339 | 389 | | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
340 | 401 | | |
341 | 402 | | |
342 | 403 | | |
| |||
750 | 811 | | |
751 | 812 | | |
752 | 813 | | |
753 | | - | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
754 | 827 | | |
755 | 828 | | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
756 | 833 | | |
757 | 834 | | |
758 | 835 | | |
| |||
779 | 856 | | |
780 | 857 | | |
781 | 858 | | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
782 | 870 | | |
783 | 871 | | |
784 | 872 | | |
| |||
793 | 881 | | |
794 | 882 | | |
795 | 883 | | |
796 | | - | |
797 | | - | |
798 | | - | |
799 | | - | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
800 | 889 | | |
801 | 890 | | |
802 | | - | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
803 | 904 | | |
804 | 905 | | |
805 | 906 | | |
| |||
871 | 972 | | |
872 | 973 | | |
873 | 974 | | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
874 | 998 | | |
875 | 999 | | |
876 | 1000 | | |
| |||
980 | 1104 | | |
981 | 1105 | | |
982 | 1106 | | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
983 | 1122 | | |
984 | 1123 | | |
985 | 1124 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
497 | 497 | | |
498 | 498 | | |
499 | 499 | | |
500 | | - | |
| 500 | + | |
501 | 501 | | |
502 | 502 | | |
503 | 503 | | |
504 | 504 | | |
505 | 505 | | |
| 506 | + | |
506 | 507 | | |
507 | | - | |
508 | | - | |
509 | | - | |
510 | | - | |
511 | | - | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
512 | 517 | | |
513 | 518 | | |
514 | 519 | | |
| |||
0 commit comments