Skip to content

Commit e5d0b75

Browse files
authored
feat: per-location credentials for the S3 credential SPI (#6031)
* feat: add CometS3ScopedCredentialProvider opt-in scope-hint sub-interface Adds `CometS3ScopedCredentialProvider`, an opt-in `@Public` sub-interface of `CometS3CredentialProvider` that lets a vendor advertise the narrowest known-safe S3-key prefixes the STS session it is about to vend will authorize. The native layer (added in a follow-up commit) will use this hint to key the `object_store` cache one entry per distinct scope on a bucket instead of the single per-bucket store used today. `CometS3CredentialDispatcher` gains a `getPolicyLocationsFor(long, String, String, int)` static method: it looks up the registered provider by handle, checks `instanceof CometS3ScopedCredentialProvider`, dispatches, and normalizes a null return to an empty list. Base-interface-only providers keep today's behavior — the method returns `Collections.emptyList()` without touching the provider, which the native side reads as "no scope hint" and falls back to single-entry-per-bucket caching. The scope hint is advisory only. The correctness ground is a 403-retry wrapper inside Comet native (added later in this series): on 403 the wrapper invalidates the entry and re-fires the SPI with the actual failing path in context, so over-reporting is self-healing at the cost of one 403 per newly discovered scope boundary and under-reporting only costs extra SPI churn. `CometPublicApiSuite` gains the new type in its pinned `@Public` set; `MinioCometS3CredentialProvider` (test fixture) implements the sub-interface with an installable prefix list for the upcoming scope-aware IT scenarios; `CometS3ScopedCredentialProviderTest` covers dispatch to base vs. scoped provider, null normalization, exception propagation, and handle validation. * feat: add scope-hint JNI bridge + RetryOn403ObjectStore correctness net Second half of the scope-hint SPI landing (Layer 1 was #TBD). Introduces the native-side plumbing that Layer 3 wires into the parquet object-store cache: * jni-bridge: add `method_get_policy_locations_for` alongside the existing `ensure_initialized` / `get_credentials_for_path` static-method IDs. Signature matches the Java dispatcher entry: `(JLjava/lang/String;Ljava/lang/String;I)Ljava/util/List;` * CometS3CredentialBridge::fetch_policy_locations — thin JNI wrapper that reads a `java.util.List<String>` back from the dispatcher. A null return, a base (non-scoped) provider, or an empty list all normalize to an empty vec, which the cache layer will interpret as "single-entry-per-bucket" semantics for backward compatibility. Rewires the module-level doc-comment to describe the new scope-hint contract (advisory prefixes, S3 remains authoritative via the 403-retry safety net layered above). * RetryOn403ObjectStore (new `parquet::objectstore::retry`): correctness ground for the advisory scope hint. On a single `Error::PermissionDenied` from the wrapped store, invoke a caller-provided rebuild closure once, then retry the same operation against the rebuilt store. A second 403 propagates without a further retry. Only `PermissionDenied` (403) triggers the retry — `Unauthenticated` (401) is treated as a permanent credential-config error. Scope of interception is deliberately narrow: `put_opts`, `get_opts`, `get_ranges`, `list_with_delimiter`, `copy_opts`, `rename_opts`. Stream variants (`list`, `list_with_offset`, `delete_stream`) surface per-item errors as-is; per-item retry would require materializing the stream and there is no observed failure mode outside `get_opts`/`get_ranges` on the parquet path. `put_multipart_opts` also skips retry — a partially-uploaded multipart cannot be transparently restarted. * Eight unit tests covering: pass-through success, rebuild-then-retry, second- 403 propagation, rebuild idempotency across ops, rebuild-error surface, 401-not-retried, Send+Sync bound, composes-with-Arc<Mutex<...>>. No callers yet: Layer 3 (scope-aware cache in `parquet_support`) will construct the rebuild closure and wire wrapped stores into the registry. Compiles clean with the expected dead-code warnings on the new symbols. * feat: scope-aware object_store cache with rebuild-on-403 wiring Final layer of the scope-hint SPI landing. Wires `CometS3ScopedCredentialProvider` (Layer 1) and the JNI bridge / retry wrapper (Layer 2) into `parquet_support`'s object-store registry so that a single bucket can host multiple scoped stores, and any misdiagnosed 403 transparently falls back to a widened-scope rebuild instead of aborting the scan. Native changes -------------- * `parquet::objectstore::s3` — factor the builder into two entry points so `parquet_support` can perform its own bridge lifecycle around the cache lookup: - `try_construct_bridge(url, configs) -> Result<Option<Arc<Bridge>>>` returns `Ok(None)` when `comet.credential.provider.class` is absent (no SPI in play), `Ok(Some(_))` when the dispatcher successfully resolves, `Err(_)` on init failure. - `create_store_with_bridge(url, configs, bridge, min_ttl)` takes the pre-constructed bridge (or `None` to fall back to the AWS credential chain) and returns the raw `AmazonS3` store. The old monolithic `create_store` is deleted — the internal `test_create_store` now calls `create_store_with_bridge(&url, &configs, None, Duration::from_secs(300))` directly. * `parquet::parquet_support` — replace the per-key `HashMap<Key, Arc<dyn ObjectStore>>` with `HashMap<Key, Vec<ScopeEntry>>` so one bucket can carry disjoint prefix-scoped stores. `ScopeEntry` pairs a `Vec<String>` of prefix hints (empty ⇒ catchall) with the store produced under those hints. Three private helpers make the intent explicit: - `path_covered(path, prefixes)` — `true` when `prefixes` is empty (catchall) or any prefix is a byte-prefix of `path`. - `find_matching_scope(entries, path)` — first entry whose prefixes cover `path`; used on the read path. - `invalidate_scope(cache, key, prefixes)` — drops the matching entry before we push its catchall replacement; prunes the key when the vector empties. The rewritten `prepare_object_store_with_configs` calls `try_construct_bridge` before the cache lookup so it can also drive the scope-hint fetch (`bridge.fetch_policy_locations().unwrap_or_default()`). On a cache miss the raw store is wrapped in `RetryOn403ObjectStore` whenever a bridge is present; the rebuild closure re-runs `try_construct_bridge` + `create_store_with_bridge`, then swaps the cache entry for a catchall entry (empty prefixes), so subsequent reads on the same bucket route through the widened store without another 403. Design trade-off worth spelling out: on 403 we widen to catchall rather than requesting a fresh scope for the failing path. The alternative would require threading the failing `Path` into the rebuild closure, which the `OnceCell` "rebuild at most once" semantics do not naturally support. In exchange for the simpler correctness net we lose vendor- scope granularity on that bucket for the remainder of the process. Well-behaved providers that never overreport a scope are unaffected; overreporting providers pay a widened-cache cost once per bucket. * `ObjectStoreCacheKey` is `(String, u64, bool)` in the tree, not the 2-tuple the plan assumed — the third element is `hdfs_backend`. All three existing seed-based tests (`check_isolated_stores`, `native_s3_aliases_share_cache_and_registration_identity`, `keeps_native_file_url_separate_from_explicit_hadoop_file_routing`) now seed `Vec<ScopeEntry>` values under the unchanged 3-tuple keys. * Three new unit tests exercise the private helpers: - `path_covered_treats_empty_prefixes_as_catchall` - `find_matching_scope_returns_first_covering_entry` - `invalidate_scope_drops_matching_entries_and_prunes_empty_keys` All 60 tests in `parquet::objectstore::*` and 16 tests in `parquet::parquet_support::tests` pass locally. Docs ---- * User guide: new "Scope hints via `CometS3ScopedCredentialProvider`" section on the S3 credential providers page describing the opt-in sub-interface, when to implement it, backward-compatibility, and the advisory / 403-retry-authoritative contract. * Contributor guide: new "Scope-aware `object_store` registry" section on the SPI design page, spelling out the `Vec<ScopeEntry>` shape, why the outer cache key stayed 3-tuple rather than absorbing prefixes, and the widen-to-catchall design trade-off. Scala IT -------- `CometS3CredentialBridgeSuite` gains two Docker-tagged (Minio) scenarios against the real bridge: * `scoped provider: two reads inside the same scope share one object_store entry` — installs a single scope prefix, reads two paths under it, verifies the scope hint is fetched only once (cache reuses the `ScopeEntry`) but credentials are refetched per read. * `scoped provider: reads under disjoint scopes get separate object_store entries` — installs scope A, reads under A, swaps to scope B, reads under B; verifies a fresh scope-hint fetch was triggered (proving cache fragmentation on disjoint prefixes rather than accidental sharing). Minio does not enforce per-prefix denial out of the box, so the full overreport-then-403-recover path is proven by `native/core/src/parquet/objectstore/retry.rs`'s eight unit tests rather than an end-to-end IT. Deferred to a follow-up if per-prefix Minio user policies become available in the test base. * feat: route S3 credentials per policy location Replace the scope-hint design with CometS3LocationScopedCredentialProvider, an opt-in extension of CometS3CredentialProvider whose getPolicyLocations(bucket) lists every location in a bucket that has its own policy. For such a provider, create_store returns a LocationScopedObjectStore that is cached and registered once per bucket like any S3 store. It serves each request with the store of the longest location covering the request's path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. A location's store is built on first use from a template whose region was resolved when the bucket's store was created, so nothing blocks on the Tokio runtime inside an async read. A 403 fetches the locations again, once for all the reads routed from the same snapshot whether the fetch succeeds or fails, and retries once if the path now routes to a different location. The dispatcher returns null for a provider that does not implement the interface, without calling it, and create_store builds the same store for it as before. Otherwise the dispatcher copies the provider's list into a String[], so list code runs inside the checked JNI call. Removes RetryOn403ObjectStore, the scope-hint JNI entry point, and the multi-entry object store cache. * fix: compile CometS3CredentialBridgeSuite on Scala 2.12 Scala 2.12 cannot choose between java.util.Set.of(E) and Set.of(E...) for a single argument, so the Spark 3.4 and 3.5 builds failed to compile the suite. Use Collections.singleton instead. * fix: derive location-scoped bridges from the bucket's bridge Each location's bridge called CometS3CredentialBridge::new again with empty catalog properties, so it only reached the bucket's provider registration while create_store also passed an empty map. Once create_store forwards the fs.s3a.* map (#6023), ensureInitialized would miss the existing key and create a second provider on whichever thread first read the location, often a Tokio worker with no context class loader. Location bridges now come from CometS3CredentialBridge::for_path, which reuses the bucket bridge's handle and bucket string and creates only the path string. They share the registration whatever create_store passes, and skip an ensureInitialized round trip per location. The MinIO suite asserts that one provider instance serves every location.
1 parent 73e5236 commit e5d0b75

16 files changed

Lines changed: 1734 additions & 39 deletions

‎docs/source/about/versioning_policy.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -276,13 +276,17 @@ here, because a vendor jar built against one Comet release is loaded by another.
276276
The SPI consists of:
277277

278278
- `CometS3CredentialProvider`, the interface a vendor implements.
279+
- `CometS3LocationScopedCredentialProvider`, an optional extension of it for buckets whose credentials
280+
differ by location.
279281
- `CometS3Credentials`, the value a provider returns.
280282
- `CometS3CredentialContext` and `CometS3AccessMode`, describing the request being served.
281283

282284
Additive changes are allowed in a minor release, for example a new accessor on
283285
`CometS3CredentialContext`, because a vendor jar compiled against an earlier `1.x` continues to
284286
load and run. Any change that would break such a jar, including adding an abstract method to
285-
`CometS3CredentialProvider` without a default implementation, requires a major release.
287+
`CometS3CredentialProvider` or `CometS3LocationScopedCredentialProvider` without a default
288+
implementation, requires a major release. The same holds for changing the meaning of an existing
289+
method, such as how `CometS3LocationScopedCredentialProvider` matches a path to a location.
286290

287291
`CometS3CredentialDispatcher` is the JNI entry point Comet uses to reach a provider. It is internal
288292
despite living in the same package, and vendors must not call it.

‎docs/source/contributor-guide/s3-credential-provider-design.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,23 @@ Comet's bridge does not maintain a TTL cache, schedule refresh, or broadcast cat
7373

7474
A Comet-side cache would have to either expose a tuning knob (TTL, max size, eviction policy) and grow over time, or be hardcoded and surprise vendors whose policies disagree. The bridge intentionally has neither and forwards every call.
7575

76+
## Location-scoped credentials on the Parquet path
77+
78+
`object_store::CredentialProvider::get_credential` receives no request path, so one `AmazonS3` store presents one credential, and the process-wide `object_store` cache holds one store per `(scheme://bucket, config_hash, hdfs_backend)`. A base provider therefore gets one credential per bucket, requested with the path of the first file Comet reads. A bucket whose policies differ by location (one for `warehouse/sales`, another for `warehouse/finance`) needs a store per location and something that picks among them for each request.
79+
80+
`CometS3LocationScopedCredentialProvider` supplies that. `getPolicyLocations(bucket)` returns every location in the bucket that has its own policy, and `create_store` returns a `LocationScopedObjectStore` for the bucket instead of a plain store:
81+
82+
- It is cached and registered like any other S3 store, one per key, so later scans on the bucket share it and the cache and registry keep one store per identity. As for any store, scans that miss the cache at the same moment each build one, and the last one cached wins.
83+
- Each request is served by the store of the longest location covering its path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. Routing is per request, so a partition whose files span several locations reads each file with its own location's credential.
84+
- A location's store is an `AmazonS3` whose bridge is bound to the location itself, so the vendor sees one stable path per location. The bridge is derived from the bucket's bridge, sharing its provider registration, so creating it calls no `ensureInitialized` and loads no classes on the Tokio worker that usually creates it. It is built on first use, usually inside an async read, from an `S3StoreTemplate` that resolved the region when the bucket's store was created, so building never blocks on the Tokio runtime.
85+
- The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, so the store fetches the locations again and retries the request once if its path now routes elsewhere. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outlives it.
86+
87+
The locations come from asking for the bucket's whole list rather than which prefixes one session covers. Asking per session leaves Comet to discover the other scopes from 403s, which needs mutable per-store state and cannot route a partition that spans scopes. With the whole list up front, routing is a function of the path.
88+
89+
This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the store caches no credentials, and every request still calls `getCredentialsForPath`. What it keeps is the provider's location list and a store for each location that has been read, so it grows with the vendor's policy list, not with the paths read. The list is only fetched again after a 403, so a change that causes none is not seen until the executor builds a new store.
90+
91+
The dispatcher returns `null` for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the 403 retry, since Comet only reads through these stores.
92+
7693
## Path-specific behavior
7794

7895
`object_store::CredentialProvider` and `reqsign_core::ProvideCredential` differ in what they consume:

‎docs/source/user-guide/latest/s3-credential-providers.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,35 @@ public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) th
201201
}
202202
```
203203

204+
### Credentials per location
205+
206+
On the Parquet path, a `CometS3CredentialProvider` gets one credential per bucket: Comet requests it with the path of the first file it reads from the bucket and uses it for every file there. If your policies differ by location within a bucket, for example one policy for `warehouse/sales` and another for `warehouse/finance`, implement `CometS3LocationScopedCredentialProvider` and tell Comet where those locations are:
207+
208+
```java
209+
public final class MyLocationProvider implements CometS3LocationScopedCredentialProvider {
210+
@Override
211+
public List<String> getPolicyLocations(String bucket) throws Exception {
212+
return policyService.locationsWithPolicies(bucket); // ["warehouse/sales", "warehouse/finance"]
213+
}
214+
215+
@Override
216+
public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) throws Exception {
217+
// ctx.getPath() is a returned location with a leading slash, or "/" for the bucket root.
218+
return sessionForLocation(ctx.getBucket(), ctx.getPath(), ctx.getMode());
219+
}
220+
}
221+
```
222+
223+
Comet serves each request with the credential of the longest location that covers its path. A location covers a path when the path is the location itself or lies below it, compared one `/`-separated segment at a time, so `warehouse/sales` covers `warehouse/sales/part-0.parquet` but not `warehouse/sales_eu/part-0.parquet`. The bucket root covers every path that no returned location covers, and an empty list serves the whole bucket with the root's credential. Write locations the way `CometS3CredentialContext.getPath()` writes paths: percent-encoded, without the scheme or bucket name. A literal `%` must be written as `%25`; other characters may be left unencoded, and a leading or trailing `/` is optional. When several locations decode to the same path, Comet keeps the first.
224+
225+
Comet requests a location's credential by calling `getCredentialsForPath` with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet's native Parquet reads only; Iceberg reads call `getCredentialsForPath` as they do for any provider.
226+
227+
**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now falls under a different location, so a location added while a job runs is picked up. A location added or removed without causing a 403 is not seen until the executor creates a new store. Make `getPolicyLocations` thread-safe and independent of where it runs; it may be called on the driver or on executors.
228+
229+
**Failures.** If `getPolicyLocations` throws or returns `null`, or returns a location that is `null` or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, `.`, `..`, or contains a control character, so a URI such as `s3://bucket/a` is invalid too. Comet does not fall back to a broader credential.
230+
231+
**Backward compatibility.** Providers that implement only `CometS3CredentialProvider` are unaffected. Comet calls nothing new on them and keeps one credential per bucket, as before.
232+
204233
### Composing multiple credential backends
205234

206235
A single configured provider class is the dispatcher. If a vendor needs to route across several credential backends (per bucket, per path prefix, per tenant), the dispatch lives inside the vendor's class:

‎native/core/src/cloud/s3/credential_bridge.rs‎

Lines changed: 69 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ use crate::execution::operators::ExecutionError;
2323
use crate::jvm_bridge::{jni_new_global_ref, jni_static_call, JVMClasses};
2424
use async_trait::async_trait;
2525
use iceberg_storage_opendal::AwsCredential as IcebergAwsCredential;
26-
use jni::objects::{Global, JFieldID, JObject, JString, JValue};
26+
use jni::objects::{Global, JFieldID, JObject, JObjectArray, JString, JValue};
2727
use jni::signature::{Primitive, ReturnType};
2828
use jni::strings::JNIString;
2929
use jni::sys::jint;
@@ -64,7 +64,9 @@ pub enum AccessMode {
6464
/// Granularity: although the JVM SPI accepts `(bucket, path)`, neither
6565
/// `object_store::CredentialProvider::get_credential` nor
6666
/// `reqsign_core::ProvideCredential::provide_credential` carries a per-request path, so the
67-
/// effective identity is per-bucket (Parquet) or per-table-location (Iceberg).
67+
/// effective identity is per-bucket (Parquet) or per-table-location (Iceberg). A Parquet provider
68+
/// that implements `CometS3LocationScopedCredentialProvider` gets one bridge per policy location
69+
/// instead; see `parquet::objectstore::location_scoped`.
6870
pub struct CometS3CredentialBridge {
6971
provider_class: String,
7072
dispatch_key: String,
@@ -134,6 +136,31 @@ impl CometS3CredentialBridge {
134136
})
135137
}
136138

139+
/// Returns a bridge to the same provider registration for another path in the bucket. It
140+
/// shares this bridge's handle and bucket string and creates only the path string, so it makes
141+
/// no `ensureInitialized` call and needs no class loading on the calling thread.
142+
pub fn for_path(&self, path: impl Into<String>) -> Result<Self, ExecutionError> {
143+
let path = path.into();
144+
let path_jstr = JVMClasses::with_env(|env| -> Result<_, ExecutionError> {
145+
let p = env
146+
.new_string(&path)
147+
.map_err(|e| ExecutionError::GeneralError(format!("new_string(path): {e}")))?;
148+
Ok(Arc::new(jni_new_global_ref!(env, p).map_err(|e| {
149+
ExecutionError::GeneralError(format!("global_ref(path): {e}"))
150+
})?))
151+
})?;
152+
Ok(Self {
153+
provider_class: self.provider_class.clone(),
154+
dispatch_key: self.dispatch_key.clone(),
155+
bucket: self.bucket.clone(),
156+
path,
157+
mode: self.mode,
158+
handle: self.handle,
159+
bucket_jstr: Arc::clone(&self.bucket_jstr),
160+
path_jstr,
161+
})
162+
}
163+
137164
fn fetch_raw(&self) -> Result<RawCredentials, ExecutionError> {
138165
JVMClasses::with_env(|env| -> Result<RawCredentials, ExecutionError> {
139166
let mode = self.mode as jint;
@@ -183,6 +210,46 @@ impl CometS3CredentialBridge {
183210
})
184211
})
185212
}
213+
214+
/// Returns the bucket's policy locations when the provider implements
215+
/// `CometS3LocationScopedCredentialProvider`, or `None` for any other provider. The
216+
/// dispatcher copies the provider's list into a `String[]`, so provider code, including a lazy
217+
/// list, runs inside the checked JNI call and its exceptions come back as errors here.
218+
pub fn policy_locations(&self) -> Result<Option<Vec<String>>, ExecutionError> {
219+
JVMClasses::with_env(|env| -> Result<Option<Vec<String>>, ExecutionError> {
220+
let locations: JObject = unsafe {
221+
jni_static_call!(env,
222+
comet_s3_credential_dispatcher.get_policy_locations(
223+
self.handle,
224+
self.bucket_jstr.as_obj()
225+
) -> JObject
226+
)?
227+
};
228+
if locations.is_null() {
229+
return Ok(None);
230+
}
231+
// SAFETY: `getPolicyLocations` is declared to return `String[]`, and the dispatcher
232+
// rejects null elements, so every element is a non-null `java.lang.String`.
233+
let locations = unsafe { JObjectArray::<JObject>::from_raw(env, locations.into_raw()) };
234+
let len = locations.len(env).map_err(|e| {
235+
ExecutionError::GeneralError(format!("policy locations length: {e}"))
236+
})?;
237+
let mut out = Vec::with_capacity(len);
238+
for i in 0..len {
239+
let element = locations.get_element(env, i).map_err(|e| {
240+
ExecutionError::GeneralError(format!("policy location {i}: {e}"))
241+
})?;
242+
let element = unsafe { JString::from_raw(&*env, element.into_raw()) };
243+
let location = element.try_to_string(env).map_err(|e| {
244+
ExecutionError::GeneralError(format!("policy location {i}: {e}"))
245+
})?;
246+
// A bucket can have more locations than the local frame holds, so free each one.
247+
env.delete_local_ref(element);
248+
out.push(location);
249+
}
250+
Ok(Some(out))
251+
})
252+
}
186253
}
187254

188255
fn ensure_initialized(

0 commit comments

Comments
 (0)