Skip to content

Concurrent grants can lose an identity while the S3 gateway migrates identity.json #14

Closed

Description

@lexfrei

A grant can get lost when several BucketAccess objects are created at once. The driver logs granted bucket access, the sidecar sets accessGranted: true and writes the Secret, but the identity never shows up in /etc/iam/identities/. Its keys get 403 InvalidAccessKeyId.

The grant still does a read-modify-write of the legacy /etc/iam/identity.json (readS3Configuration / saveS3Configuration in DriverGrantBucketAccess, same on main at f859620). Since SeaweedFS 4.08 every S3 gateway reacts to a change of that file: it loads it, writes the per-identity files and renames it to identity.json.old. With two gateway replicas both of them do it. If the next grant writes identity.json while a gateway is in the middle of that, the write is lost.

Setup: SeaweedFS chrislusf/seaweedfs:4.45@sha256:fc9f76fa993ad69966ffeb2f65d0318fcae39c6f8e20cf68ef7b3a5cb97769e5, two filers, two S3 gateways. Seen with driver v0.3.1 and with a build of 0df543f.

How to reproduce:

  1. Create a BucketClaim, then three BucketAccess objects for it in one kubectl apply.
  2. Wait a minute and try the keys from each Secret.

In 3 attempts with the 0df543f build, one lost one of its three grants. With v0.3.1 on a fresh filer, two BucketAccess objects created together lost the second one.

Logs from that attempt, for the last of the three grants (0df543f build, keys and bucket names cut):

driver    22:34:51.236 "granting bucket access" user="ba-f28f3876-28b1-40c1-8f69-26107a72739a"
gateway-1 22:34:51.351 IAM change detected in /etc/iam/identity.json, reloading configuration
gateway-1 22:34:51.354 Migrating IAM configuration to multi-file layout...
gateway-2 22:34:51.355 Migrating IAM configuration to multi-file layout...
driver    22:34:51.357 "granted bucket access" user="ba-f28f3876-28b1-40c1-8f69-26107a72739a" accessKey=<redacted>
gateway-2 22:34:51.382 Failed to migrate IAM configuration to multi-file layout: rpc error: code = Unknown desc = /etc/iam/identity.json move error: fail to move /etc/iam/identity.json => /etc/iam/identity.json.old: filer: no entry is found in filer store

After that identities/ has five files for six BucketAccess objects.

I think the grant should go through the filer IAM gRPC service (SeaweedIdentityAccessManagement: CreateUser, UpdateUser, CreateAccessKey) instead of the legacy file. The filer serves it in 4.45, and it writes the per-identity file through the credential manager, so no migration runs and there is nothing to race with.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions