seaweedfs: a Bucket user added together with others can get keys that never work #4912

Describe the bug

When a Bucket gets several users in one change, one of them can end up with credentials that never work. The BucketAccess shows accessGranted: true and its Secret is there, but every S3 request with those keys gets 403 InvalidAccessKeyId. It does not fix itself.

The COSI driver writes the grant into a file that the SeaweedFS S3 gateways rewrite at the same moment, and the write can get lost. Reported upstream as seaweedfs/seaweedfs-cosi-driver#14.

Environment

• Cozystack main, SeaweedFS 4.45, COSI driver v0.3.1 and the driver build from fix(seaweedfs): keep Object Lock working for COSI credentials on SeaweedFS 4.48 #4893. Versions from v1.5.0 on ship SeaweedFS 4.31 with the same per-identity file layout, so they should behave the same.
• On-prem test cluster, tenant SeaweedFS with two S3 gateway replicas.

To Reproduce

1. Create a Bucket with one user and wait until it is ready.
2. Add three more users to spec.users in one change.
3. After a minute, use the keys from each new bucket-<name>-<user> Secret.

In 2 of 4 attempts one user was lost. On a fresh tenant, a Bucket created with two users lost the second one.

Expected behaviour

Every user whose BucketAccess is granted gets working keys.

Actual behaviour

One user's keys get 403 InvalidAccessKeyId. The other users work.

Workaround

Remove the user from spec.users, wait until its BucketAccess is gone, then add it back on its own. The user gets a new BucketAccess and new keys in the same Secret name, and they work. The application has to pick up the new Secret. I checked this on the cluster above.

Logs

From the S3 gateway at the moment of the lost grant:

Failed to migrate IAM configuration to multi-file layout: rpc error: code = Unknown desc = /etc/iam/identity.json move error: fail to move /etc/iam/identity.json => /etc/iam/identity.json.old: filer: no entry is found in filer store