Repository navigation
Commit e5d0b75
authored
feat: per-location credentials for the S3 credential SPI (#6031)
* feat: add CometS3ScopedCredentialProvider opt-in scope-hint sub-interface
Adds `CometS3ScopedCredentialProvider`, an opt-in `@Public` sub-interface of
`CometS3CredentialProvider` that lets a vendor advertise the narrowest known-safe
S3-key prefixes the STS session it is about to vend will authorize. The native
layer (added in a follow-up commit) will use this hint to key the `object_store`
cache one entry per distinct scope on a bucket instead of the single per-bucket
store used today.
`CometS3CredentialDispatcher` gains a `getPolicyLocationsFor(long, String, String, int)`
static method: it looks up the registered provider by handle, checks `instanceof
CometS3ScopedCredentialProvider`, dispatches, and normalizes a null return to an
empty list. Base-interface-only providers keep today's behavior — the method
returns `Collections.emptyList()` without touching the provider, which the native
side reads as "no scope hint" and falls back to single-entry-per-bucket caching.
The scope hint is advisory only. The correctness ground is a 403-retry wrapper
inside Comet native (added later in this series): on 403 the wrapper invalidates
the entry and re-fires the SPI with the actual failing path in context, so
over-reporting is self-healing at the cost of one 403 per newly discovered scope
boundary and under-reporting only costs extra SPI churn.
`CometPublicApiSuite` gains the new type in its pinned `@Public` set;
`MinioCometS3CredentialProvider` (test fixture) implements the sub-interface
with an installable prefix list for the upcoming scope-aware IT scenarios;
`CometS3ScopedCredentialProviderTest` covers dispatch to base vs. scoped
provider, null normalization, exception propagation, and handle validation.
* feat: add scope-hint JNI bridge + RetryOn403ObjectStore correctness net
Second half of the scope-hint SPI landing (Layer 1 was #TBD). Introduces the
native-side plumbing that Layer 3 wires into the parquet object-store cache:
* jni-bridge: add `method_get_policy_locations_for` alongside the existing
`ensure_initialized` / `get_credentials_for_path` static-method IDs. Signature
matches the Java dispatcher entry:
`(JLjava/lang/String;Ljava/lang/String;I)Ljava/util/List;`
* CometS3CredentialBridge::fetch_policy_locations — thin JNI wrapper that reads
a `java.util.List<String>` back from the dispatcher. A null return, a base
(non-scoped) provider, or an empty list all normalize to an empty vec, which
the cache layer will interpret as "single-entry-per-bucket" semantics for
backward compatibility. Rewires the module-level doc-comment to describe the
new scope-hint contract (advisory prefixes, S3 remains authoritative via the
403-retry safety net layered above).
* RetryOn403ObjectStore (new `parquet::objectstore::retry`): correctness ground
for the advisory scope hint. On a single `Error::PermissionDenied` from the
wrapped store, invoke a caller-provided rebuild closure once, then retry the
same operation against the rebuilt store. A second 403 propagates without a
further retry. Only `PermissionDenied` (403) triggers the retry —
`Unauthenticated` (401) is treated as a permanent credential-config error.
Scope of interception is deliberately narrow: `put_opts`, `get_opts`,
`get_ranges`, `list_with_delimiter`, `copy_opts`, `rename_opts`. Stream
variants (`list`, `list_with_offset`, `delete_stream`) surface per-item errors
as-is; per-item retry would require materializing the stream and there is no
observed failure mode outside `get_opts`/`get_ranges` on the parquet path.
`put_multipart_opts` also skips retry — a partially-uploaded multipart cannot
be transparently restarted.
* Eight unit tests covering: pass-through success, rebuild-then-retry, second-
403 propagation, rebuild idempotency across ops, rebuild-error surface,
401-not-retried, Send+Sync bound, composes-with-Arc<Mutex<...>>.
No callers yet: Layer 3 (scope-aware cache in `parquet_support`) will construct
the rebuild closure and wire wrapped stores into the registry. Compiles clean
with the expected dead-code warnings on the new symbols.
* feat: scope-aware object_store cache with rebuild-on-403 wiring
Final layer of the scope-hint SPI landing. Wires
`CometS3ScopedCredentialProvider` (Layer 1) and the JNI bridge / retry
wrapper (Layer 2) into `parquet_support`'s object-store registry so that a
single bucket can host multiple scoped stores, and any misdiagnosed 403
transparently falls back to a widened-scope rebuild instead of aborting the
scan.
Native changes
--------------
* `parquet::objectstore::s3` — factor the builder into two entry points so
`parquet_support` can perform its own bridge lifecycle around the cache
lookup:
- `try_construct_bridge(url, configs) -> Result<Option<Arc<Bridge>>>`
returns `Ok(None)` when `comet.credential.provider.class` is absent
(no SPI in play), `Ok(Some(_))` when the dispatcher successfully
resolves, `Err(_)` on init failure.
- `create_store_with_bridge(url, configs, bridge, min_ttl)` takes the
pre-constructed bridge (or `None` to fall back to the AWS credential
chain) and returns the raw `AmazonS3` store. The old monolithic
`create_store` is deleted — the internal `test_create_store` now calls
`create_store_with_bridge(&url, &configs, None, Duration::from_secs(300))`
directly.
* `parquet::parquet_support` — replace the per-key
`HashMap<Key, Arc<dyn ObjectStore>>` with `HashMap<Key, Vec<ScopeEntry>>`
so one bucket can carry disjoint prefix-scoped stores. `ScopeEntry` pairs
a `Vec<String>` of prefix hints (empty ⇒ catchall) with the store
produced under those hints. Three private helpers make the intent
explicit:
- `path_covered(path, prefixes)` — `true` when `prefixes` is empty
(catchall) or any prefix is a byte-prefix of `path`.
- `find_matching_scope(entries, path)` — first entry whose prefixes
cover `path`; used on the read path.
- `invalidate_scope(cache, key, prefixes)` — drops the matching entry
before we push its catchall replacement; prunes the key when the
vector empties.
The rewritten `prepare_object_store_with_configs` calls
`try_construct_bridge` before the cache lookup so it can also drive the
scope-hint fetch (`bridge.fetch_policy_locations().unwrap_or_default()`).
On a cache miss the raw store is wrapped in `RetryOn403ObjectStore`
whenever a bridge is present; the rebuild closure re-runs
`try_construct_bridge` + `create_store_with_bridge`, then swaps the
cache entry for a catchall entry (empty prefixes), so subsequent reads on
the same bucket route through the widened store without another 403.
Design trade-off worth spelling out: on 403 we widen to catchall rather
than requesting a fresh scope for the failing path. The alternative
would require threading the failing `Path` into the rebuild closure,
which the `OnceCell` "rebuild at most once" semantics do not naturally
support. In exchange for the simpler correctness net we lose vendor-
scope granularity on that bucket for the remainder of the process.
Well-behaved providers that never overreport a scope are unaffected;
overreporting providers pay a widened-cache cost once per bucket.
* `ObjectStoreCacheKey` is `(String, u64, bool)` in the tree, not the
2-tuple the plan assumed — the third element is `hdfs_backend`. All
three existing seed-based tests (`check_isolated_stores`,
`native_s3_aliases_share_cache_and_registration_identity`,
`keeps_native_file_url_separate_from_explicit_hadoop_file_routing`) now
seed `Vec<ScopeEntry>` values under the unchanged 3-tuple keys.
* Three new unit tests exercise the private helpers:
- `path_covered_treats_empty_prefixes_as_catchall`
- `find_matching_scope_returns_first_covering_entry`
- `invalidate_scope_drops_matching_entries_and_prunes_empty_keys`
All 60 tests in `parquet::objectstore::*` and 16 tests in
`parquet::parquet_support::tests` pass locally.
Docs
----
* User guide: new "Scope hints via `CometS3ScopedCredentialProvider`"
section on the S3 credential providers page describing the opt-in
sub-interface, when to implement it, backward-compatibility, and the
advisory / 403-retry-authoritative contract.
* Contributor guide: new "Scope-aware `object_store` registry" section on
the SPI design page, spelling out the `Vec<ScopeEntry>` shape, why the
outer cache key stayed 3-tuple rather than absorbing prefixes, and the
widen-to-catchall design trade-off.
Scala IT
--------
`CometS3CredentialBridgeSuite` gains two Docker-tagged (Minio) scenarios
against the real bridge:
* `scoped provider: two reads inside the same scope share one object_store
entry` — installs a single scope prefix, reads two paths under it,
verifies the scope hint is fetched only once (cache reuses the
`ScopeEntry`) but credentials are refetched per read.
* `scoped provider: reads under disjoint scopes get separate object_store
entries` — installs scope A, reads under A, swaps to scope B, reads under
B; verifies a fresh scope-hint fetch was triggered (proving cache
fragmentation on disjoint prefixes rather than accidental sharing).
Minio does not enforce per-prefix denial out of the box, so the full
overreport-then-403-recover path is proven by
`native/core/src/parquet/objectstore/retry.rs`'s eight unit tests rather
than an end-to-end IT. Deferred to a follow-up if per-prefix Minio user
policies become available in the test base.
* feat: route S3 credentials per policy location
Replace the scope-hint design with CometS3LocationScopedCredentialProvider,
an opt-in extension of CometS3CredentialProvider whose
getPolicyLocations(bucket) lists every location in a bucket that has its own
policy.
For such a provider, create_store returns a LocationScopedObjectStore that is
cached and registered once per bucket like any S3 store. It serves each
request with the store of the longest location covering the request's path,
matched one segment at a time after percent-decoding, with the bucket root as
an implicit location. A location's store is built on first use from a
template whose region was resolved when the bucket's store was created, so
nothing blocks on the Tokio runtime inside an async read. A 403 fetches the
locations again, once for all the reads routed from the same snapshot whether
the fetch succeeds or fails, and retries once if the path now routes to a
different location.
The dispatcher returns null for a provider that does not implement the
interface, without calling it, and create_store builds the same store for it
as before. Otherwise the dispatcher copies the provider's list into a
String[], so list code runs inside the checked JNI call.
Removes RetryOn403ObjectStore, the scope-hint JNI entry point, and the
multi-entry object store cache.
* fix: compile CometS3CredentialBridgeSuite on Scala 2.12
Scala 2.12 cannot choose between java.util.Set.of(E) and Set.of(E...) for a
single argument, so the Spark 3.4 and 3.5 builds failed to compile the suite.
Use Collections.singleton instead.
* fix: derive location-scoped bridges from the bucket's bridge
Each location's bridge called CometS3CredentialBridge::new again with empty
catalog properties, so it only reached the bucket's provider registration
while create_store also passed an empty map. Once create_store forwards the
fs.s3a.* map (#6023), ensureInitialized would miss the existing key and
create a second provider on whichever thread first read the location, often
a Tokio worker with no context class loader.
Location bridges now come from CometS3CredentialBridge::for_path, which
reuses the bucket bridge's handle and bucket string and creates only the
path string. They share the registration whatever create_store passes, and
skip an ensureInitialized round trip per location. The MinIO suite asserts
that one provider instance serves every location.1 parent 73e5236 commit e5d0b75
16 files changed
Lines changed: 1734 additions & 39 deletions
File tree
- docs/source
- about
- contributor-guide
- user-guide/latest
- native
- core/src
- cloud/s3
- parquet
- objectstore
- jni-bridge/src
- spark/src
- main/java/org/apache/comet/cloud/s3
- test
- java/org/apache/comet/cloud/s3
- scala/org/apache/comet
- cloud/s3
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
276 | 276 | | |
277 | 277 | | |
278 | 278 | | |
| 279 | + | |
| 280 | + | |
279 | 281 | | |
280 | 282 | | |
281 | 283 | | |
282 | 284 | | |
283 | 285 | | |
284 | 286 | | |
285 | | - | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
286 | 290 | | |
287 | 291 | | |
288 | 292 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
76 | 93 | | |
77 | 94 | | |
78 | 95 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
201 | 201 | | |
202 | 202 | | |
203 | 203 | | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
204 | 233 | | |
205 | 234 | | |
206 | 235 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
68 | 70 | | |
69 | 71 | | |
70 | 72 | | |
| |||
134 | 136 | | |
135 | 137 | | |
136 | 138 | | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
137 | 164 | | |
138 | 165 | | |
139 | 166 | | |
| |||
183 | 210 | | |
184 | 211 | | |
185 | 212 | | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
186 | 253 | | |
187 | 254 | | |
188 | 255 | | |
| |||
0 commit comments