Repository navigation
Commit 20bfcd9
fix(tracing): close M2 (debounce drop) and M3 (export-vs-flushSync) trace corruption
Two confirmed concurrency bugs in `Trace.snapshot()` and `FileExporter.export()`
that caused trace files in long-running sessions to either lose events (M2) or
show a stale terminating status instead of "crashed" (M3). See umbrella issue
#865 for the full reproduction matrix.
M2 — debounce drops events with no follow-up snapshot
`snapshot()` returns early when `snapshotPending=true` and was never
re-scheduled after the in-flight write completed. In bursty turns (an LLM
step that fires 5-10 tool calls back-to-back), 7 of 8 events would be
dropped from disk until a fresh event arrived in a future turn. If the
process exited in the gap, the burst's tail was lost permanently. Natural
reproducer fired 20/20 trials.
Fix: track `snapshotRequestedDuringPending` whenever `snapshot()` short-
circuits. In `.finally()`, if the flag is set and the trace is neither
crashed nor ending, schedule exactly one follow-up snapshot via
`queueMicrotask`. Bounded: at most one extra write per "burst → quiet"
cycle regardless of burst size.
M3 — FileExporter.export() races flushSync with no crashed guard
`endTrace()` calls `exporter.export()`, which does
writeFile(tmp) → rename(tmp, final). The writeFile of a multi-MB trace
takes 100+ms, a wide window for `flushSync` to interleave. The `crashed`
flag added in 3846387 only guarded `snapshot()`, not `export()`, so
flushSync's synchronous crashed write was overwritten by the export's
rename. Natural reproducer fired 10/10 trials with ~52 MB traces.
Fix: give `FileExporter` a private `_crashed` flag with `markCrashed()`.
Check it at entry, before the writeFile, and before the rename (drop tmp
and bail in the last case). `flushSync` iterates exporters and calls
`markCrashed()` on each `FileExporter` BEFORE its own synchronous write.
M2 companion — endTraceStarted gate
Once `endTrace()` claims the canonical write, no concurrent snapshot may
run. Without this, the M2 follow-up's queued microtask runs after the
`await snapshotPromise` in endTrace but BEFORE the root-span endTime
mutation, capturing pre-end state and racing to be last writer. Gate the
follow-up against an `endTraceStarted` flag set at endTrace entry.
M1 (architectural residual) — NOT closed in this PR
The TOCTOU between the synchronous `if (this.crashed)` check and the
asynchronous kernel `fs.rename` syscall remains. The existing `crashed`
flag from 3846387 protects against it on local SSD (microsecond rename
window). It can theoretically fire on slow/network filesystems. Documented
as a known design hazard in the umbrella issue. Reproducer kept as
`test.skip` for local experimentation.
Tests:
- `test/altimate/tracing-rename-race.test.ts` — reproducers/regressions
for all three mechanisms. M2-natural was 20/20 → now 0/20; M3-natural
was 10/10 → now 0/10. M1-natural verifies existing protection.
- `test/altimate/tracing-display-crash.test.ts:500` updated:
`flushSync then endTrace` previously asserted endTrace overwrote
flushSync's crashed status. Under the new policy that assertion is
inverted — flushSync wins.
Closes #865
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 6ad8b47 commit 20bfcd9
3 files changed
Lines changed: 502 additions & 7 deletions
File tree
- packages/opencode
- src/altimate/observability
- test/altimate
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
165 | 165 | | |
166 | 166 | | |
167 | 167 | | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
168 | 179 | | |
169 | 180 | | |
170 | 181 | | |
171 | 182 | | |
172 | 183 | | |
173 | 184 | | |
174 | 185 | | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
175 | 189 | | |
176 | 190 | | |
177 | 191 | | |
| |||
181 | 195 | | |
182 | 196 | | |
183 | 197 | | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
184 | 203 | | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
185 | 212 | | |
186 | 213 | | |
187 | 214 | | |
| |||
337 | 364 | | |
338 | 365 | | |
339 | 366 | | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
340 | 378 | | |
341 | 379 | | |
342 | 380 | | |
| |||
750 | 788 | | |
751 | 789 | | |
752 | 790 | | |
753 | | - | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
754 | 804 | | |
755 | 805 | | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
756 | 810 | | |
757 | 811 | | |
758 | 812 | | |
| |||
779 | 833 | | |
780 | 834 | | |
781 | 835 | | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
782 | 847 | | |
783 | 848 | | |
784 | 849 | | |
| |||
871 | 936 | | |
872 | 937 | | |
873 | 938 | | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
874 | 947 | | |
875 | 948 | | |
876 | 949 | | |
| |||
980 | 1053 | | |
981 | 1054 | | |
982 | 1055 | | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
983 | 1066 | | |
984 | 1067 | | |
985 | 1068 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
497 | 497 | | |
498 | 498 | | |
499 | 499 | | |
500 | | - | |
| 500 | + | |
501 | 501 | | |
502 | 502 | | |
503 | 503 | | |
504 | 504 | | |
505 | 505 | | |
| 506 | + | |
506 | 507 | | |
507 | | - | |
508 | | - | |
509 | | - | |
510 | | - | |
511 | | - | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
512 | 517 | | |
513 | 518 | | |
514 | 519 | | |
| |||
0 commit comments