Repository navigation
seaweedfs-cosi: BucketAccess reports accessGranted before the S3 gateway accepts the key #4690
Description
A BucketAccess reports accessGranted before the SeaweedFS S3 gateway accepts its key, so the first request with the new credentials can fail with The access key ID you provided does not exist in our records.
cosi-driver v0.3.1 DriverGrantBucketAccess (pkg/driver/provisioner.go) only rewrites /etc/iam/identity.json on the filer and returns. The COSI sidecar then writes the Secret and sets accessGranted. The S3 gateway (SeaweedFS 4.45) picks the change up asynchronously from a filer metadata subscription (weed/s3api/auth_credentials_subscribe.go, onIamConfigChange). In the cozystack chart the driver writes through seaweedfs-filer (2 replicas) while the gateways follow seaweedfs-filer-client, so the change can need a filer-to-filer hop before a gateway sees it. Nothing waits for that.
Seen in e2e: job 109396174094 (two of three fresh keys worked, the third was rejected 2s after the step began) and job 109130131391 (rejected 1.25s after the BucketAccess assert finished). The bucket suite works around it with a bounded probe (#4580).
Fix shape: have the driver confirm the gateway loaded the identity before returning from DriverGrantBucketAccess (for example, a signed request with the new key against the S3 endpoint, retried briefly), so accessGranted means usable. Tenants writing right after provisioning hit the same window today.
Activity
lexfrei commentedon Oct 10, 2026
Another variant, where the key is lost for good rather than late. In e2e run 38070340996 (2026-10-10, bucket suite,
verify-s3-access) the COSI driver issued the key for BucketAccessbucket-test-adminat 17:43:14.639. Two milliseconds later both S3 pods ran the IAM migration to the multi-file layout at the same time, each doing an AtomicRename of/etc/iam/identity.jsontoidentity.json.old, and one of them failed withidentity.json not found. After that both gateways loaded 2 access keys while 3 had been issued, and they never re-read the config, so the key stayed rejected past the 60s probe from #4580. A retry in the driver would not save this one: the identity has to survive the concurrent migration, or only one gateway should run it.