Fix flush race: seal writer on shutdown so in-flight writes aren't lost#284
Merged
Conversation
The final flush at shutdown could skip (and permanently lose) a file that was being written concurrently. `FilesOrchestrator` hides files in `activeWrites` from the reader so the periodic upload worker never reads a half-written file. That's correct mid-session — the file is picked up on the next cycle — but at shutdown there is no next cycle: the worker is stopped right after the flush, so a file still in `activeWrites` during the final enumeration is stranded on disk, never uploaded. The window was a write submitted concurrently with the flush. Writes submitted before flush were already safe (serial writer queue + barrier in `closeCurrentFile`). Fix: - FileWriter.stop(): permanently seals the writer — drains in-flight writes via the serial-queue barrier, finalizes and closes the current file, then rejects all subsequent writes (logging the dropped event, with its encoded payload, for debugging). - FeatureStoreAndUpload.stop(): ordered teardown — stop scheduling uploads, then seal the writer, then run the final exhaustive flush. After sealing, no file can be in `activeWrites`, so the flush enumerates a complete set. - Exporter.shutdown(): drop the redundant pre-shutdown flush; each storage's stop() now performs the complete, race-free upload. A pre-stop flush could skip a file still being written that stop() would then never upload. - DDTracer.shutdown(): no longer pre-flushes — the OTel processor shutdown drains pending spans into the exporter (verified for both Simple and Batch processors) and the providers flush spans/logs/telemetry properly. Only the cross-process RUM port flush is forwarded, extracted into flushRUM(). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
FileWriter now takes a `log: Logger` (like DataUploadWorker) instead of the global `Log`, so callers pass the logger from the top and tests can capture log output without mutating global state. Wired `configuration.logger` / `config.logger` through at all four construction sites (spans, logs, telemetry, coverage). Tests (FileWriterTests): - write after stop() is dropped and logged with the encoded payload (asserted via an injected RecordingLogger) - stop() drains in-flight async writes before sealing - after stop(), getAllReadableFiles() enumerates the complete set — nothing hidden in activeWrites, modelling the final shutdown flush Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
anmarchenko
approved these changes
Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The final flush at shutdown could skip and permanently lose a file that was being written concurrently.
FilesOrchestratorhides files currently being appended to (activeWrites) from the reader, so the periodic upload worker never reads a half-written file. That's correct mid-session — the file is picked up on the next worker cycle. But at shutdown there is no next cycle: the worker is stopped right after the flush, so any file still inactiveWritesduring the final enumeration is stranded on disk and never uploaded.The race window is a write submitted concurrently with the flush (e.g. a span ending on another thread during teardown). Writes submitted before the flush were already safe — the serial writer queue plus the barrier in
closeCurrentFile()guarantees they complete and release theiractiveWritesentry first.The previous shutdown order made this worse:
Exporter.shutdown()flushed first, then stopped the worker — so a skipped file had no chance of being rescued.Fix
FileWriter.stop()— permanently seals the writer: drains in-flight writes via the serial-queue barrier, finalizes and closes the current file, then rejects all subsequent writes. Writes after stop are logged with their encoded payload so dropped data is debuggable.FeatureStoreAndUpload.stop()— ordered teardown: (1) stop scheduling background uploads, (2) seal the writer, (3) run the final exhaustive flush. After sealing, no file can be inactiveWrites, so the flush enumerates a complete, quiescent set — nothing is skipped. Stopping uploads first also keeps the periodic worker from racing the seal.Exporter.shutdown()— dropped the redundant pre-shutdown flush; each storage'sstop()now does the complete, race-free upload.DDTracer.shutdown()— no longer pre-flushes. The OTel processor shutdown drains pending spans into the exporter, and the providers flush spans/logs/telemetry properly on their own. Only the cross-process RUM port flush is forwarded (extracted intoflushRUM()), since provider shutdown doesn't cover it.Verification
SimpleSpanProcessor(serialprocessorQueue.syncbarrier) andBatchSpanProcessor(worker.shutdown()→forceFlushdrains the buffer synchronously beforespanExporter.shutdown()). The drained writes are enqueued on the writer queue beforewriter.stop()'s barrier, so they land on disk and get uploaded.SimpleLogRecordProcessoris fully synchronous (no buffer).FileWriterTests,EventsExporterTests,SpansExporterTests,LogsExporterTests,DataUploadWorkerTestspass.🤖 Generated with Claude Code