Repository navigation
Description
Describe the bug
The daily lifecycle walker dispatches LifecycleDelete with ExpectedIdentity left nil, and the s3 server treats a nil identity as "skip the compare-and-swap". The server also does not re-evaluate the rule against the live entry before deleting. If a key is overwritten between the walker's listing and the delete RPC (the window includes the dispatch limiter wait), the new object is deleted even though it is not expired.
On a bucket without versioning the new content is removed outright. On a bucket with versioning suspended the null version, which is the new write, is deleted and a delete marker is added. On a bucket with versioning enabled the effect is a delete marker over the new version, so the data stays recoverable but the key disappears from listings.
The event-driven path is not affected: dispatch.go fills ExpectedIdentity from the matched event (toProtoIdentity(m.Identity)). Only the walk path sends nil.
Code references
Walker, request built without identity (identical in 4.45, 4.52 and master at 0dd33f5):
- https://github.com/seaweedfs/seaweedfs/blob/4.45/weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go#L59-L67
seaweedfs/weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go
Lines 59 to 67 in 0dd33f5
req := &s3_lifecycle_pb.LifecycleDeleteRequest{ Bucket: action.Bucket, ObjectPath: objectPath, VersionId: entry.VersionID, RuleHash: rh[:], ActionKind: toProtoActionKind(action.Key.ActionKind), // ExpectedIdentity intentionally nil; server bootstraps from // the live entry on this code path. }
The unit test pins the nil identity as intended behaviour:
Event path, for comparison, sends identity:
Server, identity check accepts nil unconditionally:
- 4.45: https://github.com/seaweedfs/seaweedfs/blob/4.45/weed/s3api/s3api_internal_lifecycle.go#L41 (call) and #L328-L332 (
identityMatches,want == nilreturns true) - master: and #L388-L392
if !identityMatches(computeEntryIdentity(entry), req.ExpectedIdentity) {
Server, delete without re-evaluating the rule against the live entry:
- 4.45: https://github.com/seaweedfs/seaweedfs/blob/4.45/weed/s3api/s3api_internal_lifecycle.go#L78-L91 (suspended: delete
nullversion; default:deleteUnversionedObjectWithClient) - master:
seaweedfs/weed/s3api/s3api_internal_lifecycle.go
Lines 82 to 116 in 0dd33f5
case s3_constants.VersioningSuspended: // Best-effort null delete; NotFound is benign. Route it when an // owner resolves — on the default filer an absent entry deletes as // success, and a null version that only reached the owner would // linger. Lifecycle never bypasses retention. nullDeleted := false if owner := s3a.routableWriteOwner(req.Bucket, req.ObjectPath); owner != "" { worm, lockErr := s3a.isObjectLockEnabled(req.Bucket) if lockErr == nil { code, settled := s3a.routedDeleteNullVersion(owner, req.Bucket, req.ObjectPath, worm, false, metadataOnly) if settled { if code == s3err.ErrAccessDenied { return retryLater("ACCESS_DENIED: deleteNullVersion: object lock"), nil } if code != s3err.ErrNone { return retryLater("TRANSPORT_ERROR: routedDeleteNullVersion: " + s3err.GetAPIError(code).Code), nil } nullDeleted = true } } } if !nullDeleted { if err := s3a.deleteSpecificObjectVersion(ctx, req.Bucket, req.ObjectPath, "null", metadataOnly); err != nil { if !errors.Is(err, filer_pb.ErrNotFound) && !errors.Is(err, ErrVersionNotFound) { return retryLater("TRANSPORT_ERROR: deleteNullVersion: " + err.Error()), nil } } } if _, err := s3a.createDeleteMarker(req.Bucket, req.ObjectPath); err != nil { return retryLater("TRANSPORT_ERROR: createDeleteMarker: " + err.Error()), nil } return done(), nil default: err := s3a.WithFilerClient(false, func(c filer_pb.SeaweedFilerClient) error { return s3a.deleteUnversionedObjectWithClient(ctx, c, req.Bucket, req.ObjectPath, metadataOnly)
To reproduce
- Unversioned bucket with a rule
Expiration: { Days: 1 }and an objectlatestolder than one day. - Run the
s3_lifecyclejob so the daily walker listslatestas expired. - Between the listing and the
LifecycleDeleteRPC (easiest with a low dispatch rate limit, or a breakpoint inWalkerDispatcher.Dispatchbefored.Client.LifecycleDelete),PUTa newlatest. - The new
latestis deleted. Expected:NOOP_RESOLVED/STALE_IDENTITY, as on the event path.
Expected behavior
The walker captures the identity of the entry it listed (mtime, size, head fid, extended hash, the same fields computeEntryIdentity uses) and sends it as ExpectedIdentity, so an entry that changed after listing resolves as STALE_IDENTITY instead of being deleted. Alternatively, the server re-evaluates the rule's age condition against the live entry when no identity is given.
Impact
Any workload that rewrites a fixed key (a "latest" pointer, a manifest, a state file) under an expiration rule can lose the fresh copy. Downstream we keep lifecycle enforcement off by default until the walk path carries identity.
Version
4.45 (traced in source; the steps above were not run live), unchanged in 4.52 and master at 0dd33f5.
Activity
fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)