Skip to content

Commit 68c4e9b

Browse files
fix: refresh S3 policy locations when a location's credential fails (#6223)
* feat: add CometS3ScopedCredentialProvider opt-in scope-hint sub-interface Adds `CometS3ScopedCredentialProvider`, an opt-in `@Public` sub-interface of `CometS3CredentialProvider` that lets a vendor advertise the narrowest known-safe S3-key prefixes the STS session it is about to vend will authorize. The native layer (added in a follow-up commit) will use this hint to key the `object_store` cache one entry per distinct scope on a bucket instead of the single per-bucket store used today. `CometS3CredentialDispatcher` gains a `getPolicyLocationsFor(long, String, String, int)` static method: it looks up the registered provider by handle, checks `instanceof CometS3ScopedCredentialProvider`, dispatches, and normalizes a null return to an empty list. Base-interface-only providers keep today's behavior — the method returns `Collections.emptyList()` without touching the provider, which the native side reads as "no scope hint" and falls back to single-entry-per-bucket caching. The scope hint is advisory only. The correctness ground is a 403-retry wrapper inside Comet native (added later in this series): on 403 the wrapper invalidates the entry and re-fires the SPI with the actual failing path in context, so over-reporting is self-healing at the cost of one 403 per newly discovered scope boundary and under-reporting only costs extra SPI churn. `CometPublicApiSuite` gains the new type in its pinned `@Public` set; `MinioCometS3CredentialProvider` (test fixture) implements the sub-interface with an installable prefix list for the upcoming scope-aware IT scenarios; `CometS3ScopedCredentialProviderTest` covers dispatch to base vs. scoped provider, null normalization, exception propagation, and handle validation. * feat: add scope-hint JNI bridge + RetryOn403ObjectStore correctness net Second half of the scope-hint SPI landing (Layer 1 was #TBD). Introduces the native-side plumbing that Layer 3 wires into the parquet object-store cache: * jni-bridge: add `method_get_policy_locations_for` alongside the existing `ensure_initialized` / `get_credentials_for_path` static-method IDs. Signature matches the Java dispatcher entry: `(JLjava/lang/String;Ljava/lang/String;I)Ljava/util/List;` * CometS3CredentialBridge::fetch_policy_locations — thin JNI wrapper that reads a `java.util.List<String>` back from the dispatcher. A null return, a base (non-scoped) provider, or an empty list all normalize to an empty vec, which the cache layer will interpret as "single-entry-per-bucket" semantics for backward compatibility. Rewires the module-level doc-comment to describe the new scope-hint contract (advisory prefixes, S3 remains authoritative via the 403-retry safety net layered above). * RetryOn403ObjectStore (new `parquet::objectstore::retry`): correctness ground for the advisory scope hint. On a single `Error::PermissionDenied` from the wrapped store, invoke a caller-provided rebuild closure once, then retry the same operation against the rebuilt store. A second 403 propagates without a further retry. Only `PermissionDenied` (403) triggers the retry — `Unauthenticated` (401) is treated as a permanent credential-config error. Scope of interception is deliberately narrow: `put_opts`, `get_opts`, `get_ranges`, `list_with_delimiter`, `copy_opts`, `rename_opts`. Stream variants (`list`, `list_with_offset`, `delete_stream`) surface per-item errors as-is; per-item retry would require materializing the stream and there is no observed failure mode outside `get_opts`/`get_ranges` on the parquet path. `put_multipart_opts` also skips retry — a partially-uploaded multipart cannot be transparently restarted. * Eight unit tests covering: pass-through success, rebuild-then-retry, second- 403 propagation, rebuild idempotency across ops, rebuild-error surface, 401-not-retried, Send+Sync bound, composes-with-Arc<Mutex<...>>. No callers yet: Layer 3 (scope-aware cache in `parquet_support`) will construct the rebuild closure and wire wrapped stores into the registry. Compiles clean with the expected dead-code warnings on the new symbols. * feat: scope-aware object_store cache with rebuild-on-403 wiring Final layer of the scope-hint SPI landing. Wires `CometS3ScopedCredentialProvider` (Layer 1) and the JNI bridge / retry wrapper (Layer 2) into `parquet_support`'s object-store registry so that a single bucket can host multiple scoped stores, and any misdiagnosed 403 transparently falls back to a widened-scope rebuild instead of aborting the scan. Native changes -------------- * `parquet::objectstore::s3` — factor the builder into two entry points so `parquet_support` can perform its own bridge lifecycle around the cache lookup: - `try_construct_bridge(url, configs) -> Result<Option<Arc<Bridge>>>` returns `Ok(None)` when `comet.credential.provider.class` is absent (no SPI in play), `Ok(Some(_))` when the dispatcher successfully resolves, `Err(_)` on init failure. - `create_store_with_bridge(url, configs, bridge, min_ttl)` takes the pre-constructed bridge (or `None` to fall back to the AWS credential chain) and returns the raw `AmazonS3` store. The old monolithic `create_store` is deleted — the internal `test_create_store` now calls `create_store_with_bridge(&url, &configs, None, Duration::from_secs(300))` directly. * `parquet::parquet_support` — replace the per-key `HashMap<Key, Arc<dyn ObjectStore>>` with `HashMap<Key, Vec<ScopeEntry>>` so one bucket can carry disjoint prefix-scoped stores. `ScopeEntry` pairs a `Vec<String>` of prefix hints (empty ⇒ catchall) with the store produced under those hints. Three private helpers make the intent explicit: - `path_covered(path, prefixes)` — `true` when `prefixes` is empty (catchall) or any prefix is a byte-prefix of `path`. - `find_matching_scope(entries, path)` — first entry whose prefixes cover `path`; used on the read path. - `invalidate_scope(cache, key, prefixes)` — drops the matching entry before we push its catchall replacement; prunes the key when the vector empties. The rewritten `prepare_object_store_with_configs` calls `try_construct_bridge` before the cache lookup so it can also drive the scope-hint fetch (`bridge.fetch_policy_locations().unwrap_or_default()`). On a cache miss the raw store is wrapped in `RetryOn403ObjectStore` whenever a bridge is present; the rebuild closure re-runs `try_construct_bridge` + `create_store_with_bridge`, then swaps the cache entry for a catchall entry (empty prefixes), so subsequent reads on the same bucket route through the widened store without another 403. Design trade-off worth spelling out: on 403 we widen to catchall rather than requesting a fresh scope for the failing path. The alternative would require threading the failing `Path` into the rebuild closure, which the `OnceCell` "rebuild at most once" semantics do not naturally support. In exchange for the simpler correctness net we lose vendor- scope granularity on that bucket for the remainder of the process. Well-behaved providers that never overreport a scope are unaffected; overreporting providers pay a widened-cache cost once per bucket. * `ObjectStoreCacheKey` is `(String, u64, bool)` in the tree, not the 2-tuple the plan assumed — the third element is `hdfs_backend`. All three existing seed-based tests (`check_isolated_stores`, `native_s3_aliases_share_cache_and_registration_identity`, `keeps_native_file_url_separate_from_explicit_hadoop_file_routing`) now seed `Vec<ScopeEntry>` values under the unchanged 3-tuple keys. * Three new unit tests exercise the private helpers: - `path_covered_treats_empty_prefixes_as_catchall` - `find_matching_scope_returns_first_covering_entry` - `invalidate_scope_drops_matching_entries_and_prunes_empty_keys` All 60 tests in `parquet::objectstore::*` and 16 tests in `parquet::parquet_support::tests` pass locally. Docs ---- * User guide: new "Scope hints via `CometS3ScopedCredentialProvider`" section on the S3 credential providers page describing the opt-in sub-interface, when to implement it, backward-compatibility, and the advisory / 403-retry-authoritative contract. * Contributor guide: new "Scope-aware `object_store` registry" section on the SPI design page, spelling out the `Vec<ScopeEntry>` shape, why the outer cache key stayed 3-tuple rather than absorbing prefixes, and the widen-to-catchall design trade-off. Scala IT -------- `CometS3CredentialBridgeSuite` gains two Docker-tagged (Minio) scenarios against the real bridge: * `scoped provider: two reads inside the same scope share one object_store entry` — installs a single scope prefix, reads two paths under it, verifies the scope hint is fetched only once (cache reuses the `ScopeEntry`) but credentials are refetched per read. * `scoped provider: reads under disjoint scopes get separate object_store entries` — installs scope A, reads under A, swaps to scope B, reads under B; verifies a fresh scope-hint fetch was triggered (proving cache fragmentation on disjoint prefixes rather than accidental sharing). Minio does not enforce per-prefix denial out of the box, so the full overreport-then-403-recover path is proven by `native/core/src/parquet/objectstore/retry.rs`'s eight unit tests rather than an end-to-end IT. Deferred to a follow-up if per-prefix Minio user policies become available in the test base. * feat: route S3 credentials per policy location Replace the scope-hint design with CometS3LocationScopedCredentialProvider, an opt-in extension of CometS3CredentialProvider whose getPolicyLocations(bucket) lists every location in a bucket that has its own policy. For such a provider, create_store returns a LocationScopedObjectStore that is cached and registered once per bucket like any S3 store. It serves each request with the store of the longest location covering the request's path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. A location's store is built on first use from a template whose region was resolved when the bucket's store was created, so nothing blocks on the Tokio runtime inside an async read. A 403 fetches the locations again, once for all the reads routed from the same snapshot whether the fetch succeeds or fails, and retries once if the path now routes to a different location. The dispatcher returns null for a provider that does not implement the interface, without calling it, and create_store builds the same store for it as before. Otherwise the dispatcher copies the provider's list into a String[], so list code runs inside the checked JNI call. Removes RetryOn403ObjectStore, the scope-hint JNI entry point, and the multi-entry object store cache. * fix: compile CometS3CredentialBridgeSuite on Scala 2.12 Scala 2.12 cannot choose between java.util.Set.of(E) and Set.of(E...) for a single argument, so the Spark 3.4 and 3.5 builds failed to compile the suite. Use Collections.singleton instead. * fix: derive location-scoped bridges from the bucket's bridge Each location's bridge called CometS3CredentialBridge::new again with empty catalog properties, so it only reached the bucket's provider registration while create_store also passed an empty map. Once create_store forwards the fs.s3a.* map (#6023), ensureInitialized would miss the existing key and create a second provider on whichever thread first read the location, often a Tokio worker with no context class loader. Location bridges now come from CometS3CredentialBridge::for_path, which reuses the bucket bridge's handle and bucket string and creates only the path string. They share the registration whatever create_store passes, and skip an ensureInitialized round trip per location. The MinIO suite asserts that one provider instance serves every location. * fix: refresh S3 policy locations when a location's credential fails A LocationScopedObjectStore fetched the provider's locations again only after a 403. A provider with no policy for a path throws instead, and the bridge reported that as a plain Generic error. So a location added while the executors were up was never picked up when the provider vends no credential for the bucket root, and a location the provider dropped kept being used after it stopped vending it. Both stayed broken until the executors restarted. The bridge now gives its failures a CredentialProviderError source, which object_store passes through to the read unchanged. The store treats a read that fails with one like a 403: the same bounded refresh, and one retry if the path now routes to a different location. Other errors still return without a refresh. Closes #6221. --------- Co-authored-by: Steve Vaughan Jr <[email protected]>
1 parent 646ff18 commit 68c4e9b

5 files changed

Lines changed: 193 additions & 21 deletions

File tree

‎docs/source/contributor-guide/s3-credential-provider-design.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -82,13 +82,13 @@ A Comet-side cache would have to either expose a tuning knob (TTL, max size, evi
8282
- It is cached and registered like any other S3 store, one per key, so later scans on the bucket share it and the cache and registry keep one store per identity. As for any store, scans that miss the cache at the same moment each build one, and the last one cached wins.
8383
- Each request is served by the store of the longest location covering its path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. Routing is per request, so a partition whose files span several locations reads each file with its own location's credential.
8484
- A location's store is an `AmazonS3` whose bridge is bound to the location itself, so the vendor sees one stable path per location. The bridge is derived from the bucket's bridge, sharing its provider registration, so creating it calls no `ensureInitialized` and loads no classes on the Tokio worker that usually creates it. It is built on first use, usually inside an async read, from an `S3StoreTemplate` that resolved the region when the bucket's store was created, so building never blocks on the Tokio runtime.
85-
- The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, so the store fetches the locations again and retries the request once if its path now routes elsewhere. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outlives it.
85+
- The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, and so is a failure to get a location's credential, because a provider with no policy for a path throws rather than vending a credential S3 would reject. On either, the store fetches the locations again and retries the request once if its path now routes elsewhere. The bridge gives its failures a `CredentialProviderError` source, which `object_store` passes through to the read unchanged, so the store can tell them from other errors. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outlives it.
8686

8787
The locations come from asking for the bucket's whole list rather than which prefixes one session covers. Asking per session leaves Comet to discover the other scopes from 403s, which needs mutable per-store state and cannot route a partition that spans scopes. With the whole list up front, routing is a function of the path.
8888

89-
This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the store caches no credentials, and every request still calls `getCredentialsForPath`. What it keeps is the provider's location list and a store for each location that has been read, so it grows with the vendor's policy list, not with the paths read. The list is only fetched again after a 403, so a change that causes none is not seen until the executor builds a new store.
89+
This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the store caches no credentials, and every request still calls `getCredentialsForPath`. What it keeps is the provider's location list and a store for each location that has been read, so it grows with the vendor's policy list, not with the paths read. The list is only fetched again after a 403 or a credential failure, so a change that causes neither is not seen until the executor builds a new store.
9090

91-
The dispatcher returns `null` for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the 403 retry, since Comet only reads through these stores.
91+
The dispatcher returns `null` for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the retry, since Comet only reads through these stores.
9292

9393
## Path-specific behavior
9494

‎docs/source/user-guide/latest/s3-credential-providers.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -222,7 +222,7 @@ Comet serves each request with the credential of the longest location that cover
222222

223223
Comet requests a location's credential by calling `getCredentialsForPath` with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet's native Parquet reads only; Iceberg reads call `getCredentialsForPath` as they do for any provider.
224224

225-
**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now falls under a different location, so a location added while a job runs is picked up. A location added or removed without causing a 403 is not seen until the executor creates a new store. Make `getPolicyLocations` thread-safe and independent of where it runs; it may be called on the driver or on executors.
225+
**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, or because `getCredentialsForPath` threw for the location Comet sent it to, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now falls under a different location. So a location added while a job runs is picked up even when you vend no credential for the bucket root, and a location you drop stops being used once its credential fails. A location added or removed without a read failing on it is not seen until the executor creates a new store. Make `getPolicyLocations` thread-safe and independent of where it runs; it may be called on the driver or on executors.
226226

227227
**Failures.** If `getPolicyLocations` throws or returns `null`, or returns a location that is `null` or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, `.`, `..`, or contains a control character, so a URI such as `s3://bucket/a` is invalid too. Comet does not fall back to a broader credential.
228228

‎native/core/src/cloud/s3/credential_bridge.rs‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -339,14 +339,29 @@ struct RawCredentials {
339339
expiration_epoch_millis: i64,
340340
}
341341

342+
/// The bridge could not get a credential from the provider, as opposed to S3 rejecting one. It is
343+
/// the source of the error `get_credential` returns, which `object_store` passes through to the
344+
/// read unchanged. A `LocationScopedObjectStore` treats it like a 403, because a provider that has
345+
/// no policy for a location throws, and that can mean the location changed since its snapshot.
346+
#[derive(Debug)]
347+
pub(crate) struct CredentialProviderError(pub(crate) String);
348+
349+
impl fmt::Display for CredentialProviderError {
350+
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
351+
f.write_str(&self.0)
352+
}
353+
}
354+
355+
impl std::error::Error for CredentialProviderError {}
356+
342357
#[async_trait]
343358
impl CredentialProvider for CometS3CredentialBridge {
344359
type Credential = AwsCredential;
345360

346361
async fn get_credential(&self) -> object_store::Result<Arc<AwsCredential>> {
347362
let raw = self.fetch_raw().map_err(|e| object_store::Error::Generic {
348363
store: "S3",
349-
source: e.to_string().into(),
364+
source: Box::new(CredentialProviderError(e.to_string())),
350365
})?;
351366
Ok(Arc::new(AwsCredential {
352367
key_id: raw.access_key_id,

0 commit comments

Comments
 (0)