Repository navigation
Commit c0fdab2
fix(observability): a failing trace disk no longer stalls the server
When a trace file write failed, the sink put the whole backlog back in
its buffer. Every later push then retried the full backlog, which took
about 40% of the event loop after a few minutes of a failed disk. At
about 120k records, the unshift threw RangeError, lost the backlog, and
stopped the timed flush.
Now a failed write drops the rest of its batch and counts it. The next
flush logs the count once as a warning. The buffer stays at or below one
batch, and writes continue when the disk recovers.
Also fix the documented trace batch window default: it is 1000 ms, not
200 ms.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent a21b42c commit c0fdab2
3 files changed
Lines changed: 76 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
584 | 584 | | |
585 | 585 | | |
586 | 586 | | |
587 | | - | |
| 587 | + | |
588 | 588 | | |
589 | 589 | | |
590 | 590 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
| 17 | + | |
16 | 18 | | |
17 | 19 | | |
18 | 20 | | |
| |||
376 | 378 | | |
377 | 379 | | |
378 | 380 | | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
379 | 435 | | |
380 | 436 | | |
381 | 437 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
371 | 371 | | |
372 | 372 | | |
373 | 373 | | |
| 374 | + | |
| 375 | + | |
374 | 376 | | |
375 | 377 | | |
376 | 378 | | |
| |||
407 | 409 | | |
408 | 410 | | |
409 | 411 | | |
410 | | - | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
411 | 416 | | |
412 | 417 | | |
413 | 418 | | |
| |||
419 | 424 | | |
420 | 425 | | |
421 | 426 | | |
422 | | - | |
| 427 | + | |
423 | 428 | | |
424 | 429 | | |
| 430 | + | |
425 | 431 | | |
426 | 432 | | |
427 | 433 | | |
428 | 434 | | |
429 | 435 | | |
430 | | - | |
431 | | - | |
432 | | - | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
437 | 447 | | |
438 | 448 | | |
439 | 449 | | |
| |||
0 commit comments