Skip to content

Commit 0386e80

Browse files
t3dotggclaude
andcommitted
fix(shared): a slow trace write no longer drops records after 1,024
While a write was pending, the trace sink kept at most 1,024 records, about six seconds of spans with seven agents streaming. The spans that time a stall, such as a long sql.transaction or server.eventLoop.stall, end when it does, so a slow disk dropped the records that explain it. The backlog is now bounded by bytes only: one trace file, at least 4 MiB. The flush loop waits with Effect.timeoutOption instead of a raceFirst that needed a lint suppression. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 9ce1188 commit 0386e80

3 files changed

Lines changed: 24 additions & 23 deletions

File tree

‎docs/internals/performance-regressions.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,8 +92,9 @@ identity so polling does not invalidate subscribers.
9292

9393
Local trace records and provider event logs use asynchronous, serialized file writes and rotation.
9494
Batching synchronous appends still stalls the server event loop on slow storage, especially on
95-
Windows. Trace producers keep a bounded next batch and drop excess records under disk pressure;
96-
provider logging retains backpressure. Shutdown drains accepted writes before closing their scope.
95+
Windows. While a trace write is slow, producers keep up to one trace file of records, so the spans
96+
that time a stall survive it; records drop only while the disk stays stuck. Provider logging
97+
retains backpressure. Shutdown drains accepted writes before closing their scope.
9798

9899
Relay awareness queues one pending publication per thread and retains a follow-up when the thread
99100
changes during a send. Transcript deltas and tool output do not enqueue awareness work. Run,

‎packages/shared/src/observability.test.ts‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -348,7 +348,7 @@ describe("observability", () => {
348348
const closed = yield* Ref.make(false);
349349
const sink = yield* makeTraceSink({
350350
filePath: tracePath,
351-
maxBytes: 1_000_000,
351+
maxBytes: 5_000_000,
352352
maxFiles: 2,
353353
batchWindowMs: 10_000,
354354
}).pipe(
@@ -362,12 +362,16 @@ describe("observability", () => {
362362
}),
363363
}),
364364
);
365-
for (let index = 0; index < 256; index++) sink.push(makeRecord("first", String(index)));
365+
sink.push(makeRecord("first"));
366366
const flushing = yield* Effect.forkChild(sink.flush);
367367
yield* Deferred.await(started);
368-
// This burst runs while the OS write is held, without waiting on the disk.
368+
// These pushes return while the write is held. The backlog keeps one
369+
// trace file of records: every small one, then large ones until the
370+
// next would not fit.
369371
for (let index = 0; index < 2_000; index++)
370-
sink.push(makeRecord("next", String(index)));
372+
sink.push(makeRecord("small", String(index)));
373+
for (let index = 0; index < 3; index++)
374+
sink.push({ ...makeRecord("large"), attributes: { payload: "x".repeat(2_000_000) } });
371375
const closing = yield* Effect.forkChild(
372376
sink.close().pipe(Effect.tap(() => Ref.set(closed, true))),
373377
);
@@ -377,10 +381,9 @@ describe("observability", () => {
377381
yield* Fiber.join(flushing);
378382
yield* Fiber.join(closing);
379383
const records = yield* readTraceRecords(tracePath);
380-
assert.equal(records.length, 1_280);
381384
assert.deepStrictEqual(
382-
records.slice(0, 256).map((record) => record.name),
383-
Array.from({ length: 256 }, () => "first"),
385+
records.map((record) => record.name),
386+
["first", ...Array.from({ length: 2_000 }, () => "small"), "large", "large"],
384387
);
385388
assert.isTrue(yield* Ref.get(closed));
386389
}),

‎packages/shared/src/observability.ts‎

Lines changed: 11 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -374,8 +374,10 @@ export const makeTraceSink = Effect.fn("makeTraceSink")(function* (options: Trac
374374

375375
let buffer: Array<{ readonly text: string; readonly bytes: number }> = [];
376376
let bufferBytes = 0;
377+
// While a write is slow, push keeps up to one trace file of records (at least
378+
// 4 MiB), so the spans that time a stall still reach the file. Only a disk
379+
// that stays stuck drops records, instead of growing memory or blocking push.
377380
const maxBufferBytes = Math.max(options.maxBytes, 4 * 1024 * 1024);
378-
const maxBufferRecords = FLUSH_BUFFER_THRESHOLD * 4;
379381
const flushRequests = yield* Queue.sliding<void>(1);
380382
const flushSemaphore = yield* Semaphore.make(1);
381383
// A failure episode starts at the first dropped record and ends when a flush
@@ -401,12 +403,8 @@ export const makeTraceSink = Effect.fn("makeTraceSink")(function* (options: Trac
401403
let persistedCount = 0;
402404

403405
while (persistedCount < records.length) {
406+
// push drops any record larger than a file, so the first one always fits.
404407
const firstRecordBytes = records[persistedCount]!.bytes;
405-
if (firstRecordBytes > options.maxBytes) {
406-
persistedCount += 1;
407-
continue;
408-
}
409-
410408
let nextIndex = persistedCount + 1;
411409
let chunkBytes = firstRecordBytes;
412410
while (nextIndex < records.length) {
@@ -486,11 +484,12 @@ export const makeTraceSink = Effect.fn("makeTraceSink")(function* (options: Trac
486484

487485
yield* Effect.addFinalizer(() => flush.pipe(Effect.ignore));
488486
yield* Effect.forkScoped(
489-
// @effect-diagnostics-next-line raceFirstWithSleepToTimeout:off -- races a threshold signal against the batch interval, not a timeout
490-
Effect.raceFirst(
491-
Queue.take(flushRequests),
492-
Effect.sleep(`${options.batchWindowMs} millis`),
493-
).pipe(Effect.andThen(flush), Effect.forever),
487+
// Flush when push fills a batch or the batch window passes.
488+
Queue.take(flushRequests).pipe(
489+
Effect.timeoutOption(`${options.batchWindowMs} millis`),
490+
Effect.andThen(flush),
491+
Effect.forever,
492+
),
494493
);
495494

496495
return {
@@ -500,9 +499,7 @@ export const makeTraceSink = Effect.fn("makeTraceSink")(function* (options: Trac
500499
const text = `${JSON.stringify(record)}\n`;
501500
const bytes = Buffer.byteLength(text);
502501
if (bytes > options.maxBytes) return;
503-
// The disk can be arbitrarily slow. Keep only a bounded next batch,
504-
// rather than stalling the producer or retaining unlimited trace data.
505-
if (bufferBytes + bytes > maxBufferBytes || buffer.length >= maxBufferRecords) {
502+
if (bufferBytes + bytes > maxBufferBytes) {
506503
droppedCount++;
507504
return;
508505
}

0 commit comments

Comments
 (0)