Skip to content

Mito may delete SSTs referenced by a stored manifest delta when the save ends with RateLimited #9897

Closed

Description

@fengjiachun

What type of bug is this?

Data corruption

What subsystems are affected?

Storage Engine

Minimal reproduce step

Not reproduced; the sequence below follows from the code paths named. It applies to any deployment that keeps data and manifests on object storage (S3, GCS, OSS, Azure Blob), with any WAL.

  1. A flush or compaction saves a manifest delta. The save is an unconditional write_with (storage/utils.rs) through the object store's RetryLayer (object-store/src/util.rs).
  2. The first PUT is stored by the server, but the client sees a temporary error (timeout, connection reset or 5xx), so the retry layer sends the PUT again. The retries are throttled (429, or S3 SlowDown), and the call ends with ErrorKind::RateLimited.
  3. may_have_persisted_manifest_update treats RateLimited as proof that nothing was written (error.rs), so the flush (flush.rs) or compaction (compaction/task.rs) deletes the new SSTs.
  4. Before the next checkpoint (10 versions by default), the region is reopened, migrated, or a follower syncs. Opening replays every delta after the checkpoint, including the stored one.

What did you expect to see?

An error that can be returned after a retried write was already stored is not treated as proof that the write did not happen, and the SSTs a stored delta references are kept.

What did you see instead?

The reopened region's manifest references SST files that were deleted, so reads fail on the missing files. For a flush, the stored delta also advances flushed_entry_id, so if no later flush succeeded, the WAL entries it covers are skipped at replay and the data is lost. For a compaction, the delta removes the input files and adds outputs that no longer exist.

A possible fix: drop RateLimited from the "not persisted" list (it is the only kind in that list that the retry layer retries, so it can come back after a stored attempt). The cost is a few orphan SSTs when a throttled save really failed, which GC can reclaim.

What operating system did you use?

Any

What version of GreptimeDB did you use?

main (bb3f07b)

Relevant log output and stack trace

None.

Activity

  1. added a commit that references this issue on Oct 23, 2026
    1cb01d0
  2. added 4 commits that reference this issue on Oct 27, 2026
    70a4ecd
    cc14884
    c0eeaea
    badceec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory Bugs

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions