Repository navigation
Commit 1cb01d0
fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898)
* fix(mito2): keep SSTs when a manifest save ends with RateLimited
The object store's retry layer retries temporary errors and returns the
last attempt's error. A manifest delta PUT can be stored by the server
while the client sees a timeout, reset or 5xx, and the retries that
follow can be throttled, so the save ends with RateLimited even though
the delta exists. may_have_persisted_manifest_update treated RateLimited
as proof that nothing was written, so flush and compaction deleted the
SSTs the stored delta references. A later open or follower sync replays
the delta and reads fail on the missing files.
Treat RateLimited like other ambiguous errors: the SSTs are kept and
become orphans when the save really failed.
Fixes #9897
Signed-off-by: jeremyhi <[email protected]>
* fix(mito2): keep SSTs when a manifest save ends with PermissionDenied
The retry layer returns the last attempt's error, so a 403 from a revoked
key, a changed bucket policy or clock skew can follow a delta PUT that
the server already stored. Like RateLimited, PermissionDenied does not
prove that nothing was written, so it no longer lets flush or compaction
delete the SSTs; they become orphans when the save really failed.
The compaction test that rejects a manifest write before persistence now
injects Unsupported, which is raised before any request is sent.
Signed-off-by: jeremyhi <[email protected]>
---------
Signed-off-by: jeremyhi <[email protected]>
Co-authored-by: Yingwen <[email protected]>1 parent afbe77e commit 1cb01d0
2 files changed
Lines changed: 16 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1414 | 1414 | | |
1415 | 1415 | | |
1416 | 1416 | | |
1417 | | - | |
| 1417 | + | |
1418 | 1418 | | |
1419 | 1419 | | |
1420 | 1420 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1445 | 1445 | | |
1446 | 1446 | | |
1447 | 1447 | | |
1448 | | - | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
1449 | 1451 | | |
1450 | 1452 | | |
1451 | 1453 | | |
| |||
1459 | 1461 | | |
1460 | 1462 | | |
1461 | 1463 | | |
1462 | | - | |
1463 | 1464 | | |
1464 | 1465 | | |
1465 | 1466 | | |
1466 | | - | |
1467 | 1467 | | |
1468 | 1468 | | |
1469 | 1469 | | |
| |||
1847 | 1847 | | |
1848 | 1848 | | |
1849 | 1849 | | |
1850 | | - | |
1851 | 1850 | | |
1852 | 1851 | | |
1853 | 1852 | | |
1854 | | - | |
1855 | 1853 | | |
1856 | 1854 | | |
1857 | 1855 | | |
| |||
1864 | 1862 | | |
1865 | 1863 | | |
1866 | 1864 | | |
1867 | | - | |
1868 | | - | |
1869 | | - | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
1870 | 1877 | | |
1871 | 1878 | | |
1872 | 1879 | | |
| |||
0 commit comments