Skip to content

Fix flush race: seal writer on shutdown so in-flight writes aren't lost#284

Merged
ypopovych merged 4 commits into
mainfrom
fix/flush-race-seal-writer-on-shutdown
Jun 19, 2026
Merged

Fix flush race: seal writer on shutdown so in-flight writes aren't lost#284
ypopovych merged 4 commits into
mainfrom
fix/flush-race-seal-writer-on-shutdown

Conversation

@ypopovych

Copy link
Copy Markdown
Contributor

Problem

The final flush at shutdown could skip and permanently lose a file that was being written concurrently.

FilesOrchestrator hides files currently being appended to (activeWrites) from the reader, so the periodic upload worker never reads a half-written file. That's correct mid-session — the file is picked up on the next worker cycle. But at shutdown there is no next cycle: the worker is stopped right after the flush, so any file still in activeWrites during the final enumeration is stranded on disk and never uploaded.

The race window is a write submitted concurrently with the flush (e.g. a span ending on another thread during teardown). Writes submitted before the flush were already safe — the serial writer queue plus the barrier in closeCurrentFile() guarantees they complete and release their activeWrites entry first.

The previous shutdown order made this worse: Exporter.shutdown() flushed first, then stopped the worker — so a skipped file had no chance of being rescued.

Fix

  • FileWriter.stop() — permanently seals the writer: drains in-flight writes via the serial-queue barrier, finalizes and closes the current file, then rejects all subsequent writes. Writes after stop are logged with their encoded payload so dropped data is debuggable.
  • FeatureStoreAndUpload.stop() — ordered teardown: (1) stop scheduling background uploads, (2) seal the writer, (3) run the final exhaustive flush. After sealing, no file can be in activeWrites, so the flush enumerates a complete, quiescent set — nothing is skipped. Stopping uploads first also keeps the periodic worker from racing the seal.
  • Exporter.shutdown() — dropped the redundant pre-shutdown flush; each storage's stop() now does the complete, race-free upload.
  • DDTracer.shutdown() — no longer pre-flushes. The OTel processor shutdown drains pending spans into the exporter, and the providers flush spans/logs/telemetry properly on their own. Only the cross-process RUM port flush is forwarded (extracted into flushRUM()), since provider shutdown doesn't cover it.

Verification

  • Confirmed the OTel shutdown path drains before tearing down for both SimpleSpanProcessor (serial processorQueue.sync barrier) and BatchSpanProcessor (worker.shutdown()forceFlush drains the buffer synchronously before spanExporter.shutdown()). The drained writes are enqueued on the writer queue before writer.stop()'s barrier, so they land on disk and get uploaded.
  • SimpleLogRecordProcessor is fully synchronous (no buffer).
  • Builds; FileWriterTests, EventsExporterTests, SpansExporterTests, LogsExporterTests, DataUploadWorkerTests pass.

🤖 Generated with Claude Code

The final flush at shutdown could skip (and permanently lose) a file that
was being written concurrently. `FilesOrchestrator` hides files in
`activeWrites` from the reader so the periodic upload worker never reads a
half-written file. That's correct mid-session — the file is picked up on
the next cycle — but at shutdown there is no next cycle: the worker is
stopped right after the flush, so a file still in `activeWrites` during the
final enumeration is stranded on disk, never uploaded.

The window was a write submitted concurrently with the flush. Writes
submitted before flush were already safe (serial writer queue + barrier in
`closeCurrentFile`).

Fix:
- FileWriter.stop(): permanently seals the writer — drains in-flight writes
  via the serial-queue barrier, finalizes and closes the current file, then
  rejects all subsequent writes (logging the dropped event, with its encoded
  payload, for debugging).
- FeatureStoreAndUpload.stop(): ordered teardown — stop scheduling uploads,
  then seal the writer, then run the final exhaustive flush. After sealing,
  no file can be in `activeWrites`, so the flush enumerates a complete set.
- Exporter.shutdown(): drop the redundant pre-shutdown flush; each storage's
  stop() now performs the complete, race-free upload. A pre-stop flush could
  skip a file still being written that stop() would then never upload.
- DDTracer.shutdown(): no longer pre-flushes — the OTel processor shutdown
  drains pending spans into the exporter (verified for both Simple and Batch
  processors) and the providers flush spans/logs/telemetry properly. Only the
  cross-process RUM port flush is forwarded, extracted into flushRUM().

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ypopovych
ypopovych requested review from a team as code owners June 17, 2026 11:16
ypopovych and others added 3 commits June 17, 2026 12:28
FileWriter now takes a `log: Logger` (like DataUploadWorker) instead of the
global `Log`, so callers pass the logger from the top and tests can capture
log output without mutating global state. Wired `configuration.logger` /
`config.logger` through at all four construction sites (spans, logs,
telemetry, coverage).

Tests (FileWriterTests):
- write after stop() is dropped and logged with the encoded payload
  (asserted via an injected RecordingLogger)
- stop() drains in-flight async writes before sealing
- after stop(), getAllReadableFiles() enumerates the complete set — nothing
  hidden in activeWrites, modelling the final shutdown flush

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ypopovych
ypopovych requested a review from anmarchenko June 19, 2026 13:42
@ypopovych
ypopovych merged commit c211bea into main Jun 19, 2026
50 of 51 checks passed
@ypopovych
ypopovych deleted the fix/flush-race-seal-writer-on-shutdown branch June 19, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants