You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Mito may delete SSTs referenced by a stored manifest delta when the save ends with RateLimited #9897
Not reproduced; the sequence below follows from the code paths named. It applies to any deployment that keeps data and manifests on object storage (S3, GCS, OSS, Azure Blob), with any WAL.
A flush or compaction saves a manifest delta. The save is an unconditional write_with (storage/utils.rs) through the object store's RetryLayer (object-store/src/util.rs).
The first PUT is stored by the server, but the client sees a temporary error (timeout, connection reset or 5xx), so the retry layer sends the PUT again. The retries are throttled (429, or S3 SlowDown), and the call ends with ErrorKind::RateLimited.
may_have_persisted_manifest_update treats RateLimited as proof that nothing was written (error.rs), so the flush (flush.rs) or compaction (compaction/task.rs) deletes the new SSTs.
Before the next checkpoint (10 versions by default), the region is reopened, migrated, or a follower syncs. Opening replays every delta after the checkpoint, including the stored one.
What did you expect to see?
An error that can be returned after a retried write was already stored is not treated as proof that the write did not happen, and the SSTs a stored delta references are kept.
What did you see instead?
The reopened region's manifest references SST files that were deleted, so reads fail on the missing files. For a flush, the stored delta also advances flushed_entry_id, so if no later flush succeeded, the WAL entries it covers are skipped at replay and the data is lost. For a compaction, the delta removes the input files and adds outputs that no longer exist.
A possible fix: drop RateLimited from the "not persisted" list (it is the only kind in that list that the retry layer retries, so it can come back after a stored attempt). The cost is a few orphan SSTs when a throttled save really failed, which GC can reclaim.
Activity
fix(mito2): keep SSTs when a manifest save ends with RateLimited or P…
fix(mito2): keep SSTs when a manifest save ends with RateLimited or P…
fix(mito2): keep SSTs when a manifest save ends with RateLimited or P…
fix(mito2): keep SSTs when a manifest save ends with RateLimited or P…
fix: backport selected fixes to v1.3 (#10006)