You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Location-scoped S3 credentials: refresh the locations when a location's credential fails, not only on 403 #6212
Context.#6031 adds CometS3LocationScopedCredentialProvider: the Parquet path's LocationScopedObjectStore keeps the provider's policy locations as a snapshot and fetches them again only when S3 returns 403 (review).
Problem. A provider with no policy for the path it is asked about usually throws from getCredentialsForPath instead of returning a credential that S3 then rejects. The bridge turns that exception into object_store::Error::Generic, which does not trigger the refresh. Two ways this shows up:
The provider has no bucket-wide policy and throws for /, then adds warehouse/finance mid-job. Reads under it route to / and keep failing, so the new location is never picked up.
The provider folds warehouse/finance into warehouse and throws for the old location. Later reads under it keep failing, while a freshly built store would serve the same path from /warehouse.
The store lives as long as the executor, so both stay broken until a restart. The user guide says a location added while a job runs is picked up, and suggests keying the provider's cache on the location, which leads naturally to that throw.
Options.
(a) Treat a credential failure from a location's store like a 403: mark the bridge's errors with a dedicated error type the store can recognize, and run the same refresh and single retry.
(b) Require getCredentialsForPath to answer for / and for locations the provider has dropped, and say so in the Javadoc and user guide.
Proposal. (a). It keeps the documented behavior and a simple vendor contract, and it changes only the implementation. If we choose (b) instead, it should land before CometS3LocationScopedCredentialProvider first ships, since the versioning policy treats a change in a method's meaning as breaking.
Activity