You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Concurrent grants can lose an identity while the S3 gateway migrates identity.json #14
A grant can get lost when several BucketAccess objects are created at once. The driver logs granted bucket access, the sidecar sets accessGranted: true and writes the Secret, but the identity never shows up in /etc/iam/identities/. Its keys get 403 InvalidAccessKeyId.
The grant still does a read-modify-write of the legacy /etc/iam/identity.json (readS3Configuration / saveS3Configuration in DriverGrantBucketAccess, same on main at f859620). Since SeaweedFS 4.08 every S3 gateway reacts to a change of that file: it loads it, writes the per-identity files and renames it to identity.json.old. With two gateway replicas both of them do it. If the next grant writes identity.json while a gateway is in the middle of that, the write is lost.
Setup: SeaweedFS chrislusf/seaweedfs:4.45@sha256:fc9f76fa993ad69966ffeb2f65d0318fcae39c6f8e20cf68ef7b3a5cb97769e5, two filers, two S3 gateways. Seen with driver v0.3.1 and with a build of 0df543f.
How to reproduce:
Create a BucketClaim, then three BucketAccess objects for it in one kubectl apply.
Wait a minute and try the keys from each Secret.
In 3 attempts with the 0df543f build, one lost one of its three grants. With v0.3.1 on a fresh filer, two BucketAccess objects created together lost the second one.
Logs from that attempt, for the last of the three grants (0df543f build, keys and bucket names cut):
driver 22:34:51.236 "granting bucket access" user="ba-f28f3876-28b1-40c1-8f69-26107a72739a"
gateway-1 22:34:51.351 IAM change detected in /etc/iam/identity.json, reloading configuration
gateway-1 22:34:51.354 Migrating IAM configuration to multi-file layout...
gateway-2 22:34:51.355 Migrating IAM configuration to multi-file layout...
driver 22:34:51.357 "granted bucket access" user="ba-f28f3876-28b1-40c1-8f69-26107a72739a" accessKey=<redacted>
gateway-2 22:34:51.382 Failed to migrate IAM configuration to multi-file layout: rpc error: code = Unknown desc = /etc/iam/identity.json move error: fail to move /etc/iam/identity.json => /etc/iam/identity.json.old: filer: no entry is found in filer store
After that identities/ has five files for six BucketAccess objects.
I think the grant should go through the filer IAM gRPC service (SeaweedIdentityAccessManagement: CreateUser, UpdateUser, CreateAccessKey) instead of the legacy file. The filer serves it in 4.45, and it writes the per-identity file through the credential manager, so no migration runs and there is nothing to race with.
Activity