Skip to content

s3 lifecycle: daily walker deletes without ExpectedIdentity, so an object overwritten after listing is expired #12451

Closed

Description

@scooby87

Describe the bug

The daily lifecycle walker dispatches LifecycleDelete with ExpectedIdentity left nil, and the s3 server treats a nil identity as "skip the compare-and-swap". The server also does not re-evaluate the rule against the live entry before deleting. If a key is overwritten between the walker's listing and the delete RPC (the window includes the dispatch limiter wait), the new object is deleted even though it is not expired.

On a bucket without versioning the new content is removed outright. On a bucket with versioning suspended the null version, which is the new write, is deleted and a delete marker is added. On a bucket with versioning enabled the effect is a delete marker over the new version, so the data stays recoverable but the key disappears from listings.

The event-driven path is not affected: dispatch.go fills ExpectedIdentity from the matched event (toProtoIdentity(m.Identity)). Only the walk path sends nil.

Code references

Walker, request built without identity (identical in 4.45, 4.52 and master at 0dd33f5):

The unit test pins the nil identity as intended behaviour:

Event path, for comparison, sends identity:

Server, identity check accepts nil unconditionally:

Server, delete without re-evaluating the rule against the live entry:

  • 4.45: https://github.com/seaweedfs/seaweedfs/blob/4.45/weed/s3api/s3api_internal_lifecycle.go#L78-L91 (suspended: delete null version; default: deleteUnversionedObjectWithClient)
  • master:
    case s3_constants.VersioningSuspended:
    // Best-effort null delete; NotFound is benign. Route it when an
    // owner resolves — on the default filer an absent entry deletes as
    // success, and a null version that only reached the owner would
    // linger. Lifecycle never bypasses retention.
    nullDeleted := false
    if owner := s3a.routableWriteOwner(req.Bucket, req.ObjectPath); owner != "" {
    worm, lockErr := s3a.isObjectLockEnabled(req.Bucket)
    if lockErr == nil {
    code, settled := s3a.routedDeleteNullVersion(owner, req.Bucket, req.ObjectPath, worm, false, metadataOnly)
    if settled {
    if code == s3err.ErrAccessDenied {
    return retryLater("ACCESS_DENIED: deleteNullVersion: object lock"), nil
    }
    if code != s3err.ErrNone {
    return retryLater("TRANSPORT_ERROR: routedDeleteNullVersion: " + s3err.GetAPIError(code).Code), nil
    }
    nullDeleted = true
    }
    }
    }
    if !nullDeleted {
    if err := s3a.deleteSpecificObjectVersion(ctx, req.Bucket, req.ObjectPath, "null", metadataOnly); err != nil {
    if !errors.Is(err, filer_pb.ErrNotFound) && !errors.Is(err, ErrVersionNotFound) {
    return retryLater("TRANSPORT_ERROR: deleteNullVersion: " + err.Error()), nil
    }
    }
    }
    if _, err := s3a.createDeleteMarker(req.Bucket, req.ObjectPath); err != nil {
    return retryLater("TRANSPORT_ERROR: createDeleteMarker: " + err.Error()), nil
    }
    return done(), nil
    default:
    err := s3a.WithFilerClient(false, func(c filer_pb.SeaweedFilerClient) error {
    return s3a.deleteUnversionedObjectWithClient(ctx, c, req.Bucket, req.ObjectPath, metadataOnly)

To reproduce

  1. Unversioned bucket with a rule Expiration: { Days: 1 } and an object latest older than one day.
  2. Run the s3_lifecycle job so the daily walker lists latest as expired.
  3. Between the listing and the LifecycleDelete RPC (easiest with a low dispatch rate limit, or a breakpoint in WalkerDispatcher.Dispatch before d.Client.LifecycleDelete), PUT a new latest.
  4. The new latest is deleted. Expected: NOOP_RESOLVED / STALE_IDENTITY, as on the event path.

Expected behavior

The walker captures the identity of the entry it listed (mtime, size, head fid, extended hash, the same fields computeEntryIdentity uses) and sends it as ExpectedIdentity, so an entry that changed after listing resolves as STALE_IDENTITY instead of being deleted. Alternatively, the server re-evaluates the rule's age condition against the live entry when no identity is given.

Impact

Any workload that rewrites a fixed key (a "latest" pointer, a manifest, a state file) under an expiration rule can lose the fresh copy. Downstream we keep lifecycle enforcement off by default until the walk path carries identity.

Version

4.45 (traced in source; the steps above were not run live), unchanged in 4.52 and master at 0dd33f5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions