Repository navigation
objectstore: finish snapshot cleanup when the bucket is missing - #1973
Luiz Oliveira (laoj2) merged 2 commits into
Conversation
Bowei Du (bowei)
added
area/reliability
area/storage
kind/bug
| objects, err := s.List(ctx, bucket, prefix) | ||
| if err != nil { | ||
| if isMissingBucket(err) { | ||
| return nil |
There was a problem hiding this comment.
can we log a warning here, instead of returning silently?
There was a problem hiding this comment.
Addressed in e73bfd6. DeletePrefix now logs a warning when a missing bucket is detected during either listing or per-object deletion, including the bucket, prefix/object, and provider error. Added TestDeletePrefix_MissingBucketWarning to cover both paths for S3 and GCS and assert the warning level, bucket, and provider error.
| @@ -24,11 +24,14 @@ package objectstore | |||
|
|
|||
| import ( | |||
There was a problem hiding this comment.
can we add a functional test covering the DeleteActor path from #1970?
e.g., trying to delete an actor, whose bucket is gone still succeeds.
There was a problem hiding this comment.
Added TestDeleteActor_MissingSnapshotBucket in e73bfd6. It exercises the DeleteActor RPC with wrapped S3 NoSuchBucket and GCS ErrBucketNotExist errors injected at the object-store boundary. The cases cover a SUSPENDED actor with an external snapshot and a DELETING actor with an in-progress snapshot; each verifies the cleanup prefix, successful deletion, and a subsequent GetActor returning NotFound. The latest pr-workflow run is awaiting workflow approval (action_required).
|
Luiz Oliveira (@laoj2) could you please re-enqueue this PR in the merge queue for |
Fixes #1970
Actor deletion can get stuck when its snapshot bucket has already been removed. Treat S3
NoSuchBucketand GCSstorage.ErrBucketNotExistas completed cleanup inDeletePrefix, including when the bucket disappears after listing. Log a warning with the bucket, prefix or object, and provider error when taking this path. Other storage errors still fail cleanup, and copies from a missing source still fail.Regression coverage:
TestDeleteActor_MissingSnapshotBucketexercises the DeleteActor RPC with wrapped S3/GCS missing-bucket errors injected at the object-store boundary. It covers a SUSPENDED actor with an external snapshot and a DELETING actor with an in-progress snapshot, verifies the cleanup prefix, and confirms that GetActor returns NotFound after deletion.Validation for
e73bfd6: pr-workflow passed, including unit tests and E2E. The Actions Workflow Security Scan, GitHub Actions Scan, and CLA checks also passed.AI tools were used in preparing this PR.