Skip to content

bug: ShardWriter::close can report success after final flush failures #7770

Description

@u70b3

Problem

ShardWriter::close() is the graceful shutdown API that flushes pending data
before stopping background tasks. However, it currently waits for several
close-time completion notifications while discarding their results:

  • the final WAL flush in MemTable mode;
  • frozen MemTable/L0 flushes;
  • the final WAL flush in WAL-only mode.

Failures to send the final WAL flush request and completion channels closing
without a result are also treated as success.

Failure sequence

sequenceDiagram
    participant Caller
    participant Close as ShardWriter::close()
    participant Flusher as WAL / MemTable flusher
    participant Store as Object store
    participant Tasks as TaskExecutor

    Caller->>Close: close()
    Close->>Flusher: request final flush
    Flusher->>Store: persist pending data
    Store-->>Flusher: persistence failure
    Flusher-->>Close: Err / DurabilityResult::Failed
    Note over Close: Completion is awaited,<br/>but its value is discarded
    Close->>Tasks: shutdown_all()
    Tasks-->>Close: Ok(())
    Close-->>Caller: Ok(()) (incorrect)
    Note over Caller,Store: Caller assumes durability<br/>although data was not persisted
Loading

Impact

In WAL-only mode, a failed final WAL append leaves the batch in the in-memory
pending queue. Because close() consumes the writer, that queue is then dropped
while the caller receives Ok(()).

In MemTable mode, callers can receive a successful close result even though the
final WAL or L0 generation was not persisted.

This is a false durability acknowledgement and may cause data loss that is
invisible to the caller.

Expected behavior

ShardWriter::close() should:

  • return final WAL and MemTable flush failures;
  • preserve typed FenceReason values carried by WalFlushFailure;
  • return the first causal failure;
  • continue draining already queued work after an error;
  • always shut down and join background tasks before returning;
  • keep successful close behavior unchanged.

Acceptance criteria

  • WAL-only final persistence failures are returned to the caller.
  • MemTable final WAL persistence failures are returned to the caller.
  • Frozen MemTable/L0 failures are returned to the caller.
  • WAL channel send failures and missing completion results return contextual
    errors.
  • Earlier typed failures are not replaced by later generic errors.
  • Background tasks are shut down on both success and error paths.
  • Failure-focused tests cover all three persistence paths.
  • No public API, binding, dependency, or persistent-format changes are required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions