Skip to content

Commit 1cb01d0

Browse files
fengjiachunevenyag
andauthored
fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898)
* fix(mito2): keep SSTs when a manifest save ends with RateLimited The object store's retry layer retries temporary errors and returns the last attempt's error. A manifest delta PUT can be stored by the server while the client sees a timeout, reset or 5xx, and the retries that follow can be throttled, so the save ends with RateLimited even though the delta exists. may_have_persisted_manifest_update treated RateLimited as proof that nothing was written, so flush and compaction deleted the SSTs the stored delta references. A later open or follower sync replays the delta and reads fail on the missing files. Treat RateLimited like other ambiguous errors: the SSTs are kept and become orphans when the save really failed. Fixes #9897 Signed-off-by: jeremyhi <[email protected]> * fix(mito2): keep SSTs when a manifest save ends with PermissionDenied The retry layer returns the last attempt's error, so a 403 from a revoked key, a changed bucket policy or clock skew can follow a delta PUT that the server already stored. Like RateLimited, PermissionDenied does not prove that nothing was written, so it no longer lets flush or compaction delete the SSTs; they become orphans when the save really failed. The compaction test that rejects a manifest write before persistence now injects Unsupported, which is raised before any request is sent. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]> Co-authored-by: Yingwen <[email protected]>
1 parent afbe77e commit 1cb01d0

2 files changed

Lines changed: 16 additions & 9 deletions

File tree

‎src/mito2/src/engine/compaction_test.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1414,7 +1414,7 @@ impl object_store::layers::mock::Write for FailingManifestWriter {
14141414
) -> object_store::layers::mock::Result<()> {
14151415
if matches!(self.failure, ManifestWriteFailure::BeforeWrite) {
14161416
return Err(object_store::layers::mock::Error::new(
1417-
object_store::layers::mock::ErrorKind::PermissionDenied,
1417+
object_store::layers::mock::ErrorKind::Unsupported,
14181418
"manifest write rejected before persistence",
14191419
));
14201420
}

‎src/mito2/src/error.rs‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1445,7 +1445,9 @@ impl Error {
14451445
/// Returns whether a failed manifest update may have been persisted.
14461446
///
14471447
/// Unknown errors are treated conservatively because deleting output SSTs after an ambiguous
1448-
/// manifest write could leave the manifest referencing missing files.
1448+
/// manifest write could leave the manifest referencing missing files. The retry layer returns
1449+
/// the last attempt's error, so an error that can follow a stored attempt (throttling, or a 403
1450+
/// from a revoked key, a changed bucket policy or clock skew) never proves nothing was written.
14491451
pub(crate) fn may_have_persisted_manifest_update(&self) -> bool {
14501452
match self {
14511453
Error::UpdateManifest { .. }
@@ -1459,11 +1461,9 @@ impl Error {
14591461
ErrorKind::Unsupported
14601462
| ErrorKind::ConfigInvalid
14611463
| ErrorKind::NotFound
1462-
| ErrorKind::PermissionDenied
14631464
| ErrorKind::IsADirectory
14641465
| ErrorKind::NotADirectory
14651466
| ErrorKind::AlreadyExists
1466-
| ErrorKind::RateLimited
14671467
| ErrorKind::IsSameFile
14681468
| ErrorKind::ConditionNotMatch
14691469
| ErrorKind::RangeNotSatisfied
@@ -1847,11 +1847,9 @@ mod tests {
18471847
ErrorKind::Unsupported,
18481848
ErrorKind::ConfigInvalid,
18491849
ErrorKind::NotFound,
1850-
ErrorKind::PermissionDenied,
18511850
ErrorKind::IsADirectory,
18521851
ErrorKind::NotADirectory,
18531852
ErrorKind::AlreadyExists,
1854-
ErrorKind::RateLimited,
18551853
ErrorKind::IsSameFile,
18561854
ErrorKind::ConditionNotMatch,
18571855
ErrorKind::RangeNotSatisfied,
@@ -1864,9 +1862,18 @@ mod tests {
18641862
);
18651863
}
18661864

1867-
let error =
1868-
OpenDalSnafu {}.into_error(object_store::Error::new(ErrorKind::Unexpected, "test"));
1869-
assert!(error.may_have_persisted_manifest_update());
1865+
// The last retry can fail with these kinds even though an earlier attempt was stored.
1866+
for kind in [
1867+
ErrorKind::Unexpected,
1868+
ErrorKind::RateLimited,
1869+
ErrorKind::PermissionDenied,
1870+
] {
1871+
let error = OpenDalSnafu {}.into_error(object_store::Error::new(kind, "test"));
1872+
assert!(
1873+
error.may_have_persisted_manifest_update(),
1874+
"error kind {kind:?} should not prove the manifest was not persisted"
1875+
);
1876+
}
18701877

18711878
let region_id = RegionId::new(1, 1);
18721879
let error = RegionTruncatedSnafu { region_id }.build();

0 commit comments

Comments
 (0)