Skip to content

Commit cb1bdec

Browse files
committed
feat: route S3 credentials per policy location
Replace the scope-hint design with CometS3LocationScopedCredentialProvider, an opt-in extension of CometS3CredentialProvider whose getPolicyLocations(bucket) lists every location in a bucket that has its own policy. For such a provider, create_store returns a LocationScopedObjectStore that is cached and registered once per bucket like any S3 store. It serves each request with the store of the longest location covering the request's path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. A location's store is built on first use from a template whose region was resolved when the bucket's store was created, so nothing blocks on the Tokio runtime inside an async read. A 403 fetches the locations again, once for all the reads routed from the same snapshot whether the fetch succeeds or fails, and retries once if the path now routes to a different location. The dispatcher returns null for a provider that does not implement the interface, without calling it, and create_store builds the same store for it as before. Otherwise the dispatcher copies the provider's list into a String[], so list code runs inside the checked JNI call. Removes RetryOn403ObjectStore, the scope-hint JNI entry point, and the multi-entry object store cache.
1 parent 8a86d80 commit cb1bdec

20 files changed

Lines changed: 1625 additions & 1523 deletions

‎docs/source/about/versioning_policy.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -209,13 +209,17 @@ here, because a vendor jar built against one Comet release is loaded by another.
209209
The SPI consists of:
210210

211211
- `CometS3CredentialProvider`, the interface a vendor implements.
212+
- `CometS3LocationScopedCredentialProvider`, an optional extension of it for buckets whose credentials
213+
differ by location.
212214
- `CometS3Credentials`, the value a provider returns.
213215
- `CometS3CredentialContext` and `CometS3AccessMode`, describing the request being served.
214216

215217
Additive changes are allowed in a minor release, for example a new accessor on
216218
`CometS3CredentialContext`, because a vendor jar compiled against an earlier `1.x` continues to
217219
load and run. Any change that would break such a jar, including adding an abstract method to
218-
`CometS3CredentialProvider` without a default implementation, requires a major release.
220+
`CometS3CredentialProvider` or `CometS3LocationScopedCredentialProvider` without a default
221+
implementation, requires a major release. The same holds for changing the meaning of an existing
222+
method, such as how `CometS3LocationScopedCredentialProvider` matches a path to a location.
219223

220224
`CometS3CredentialDispatcher` is the JNI entry point Comet uses to reach a provider. It is internal
221225
despite living in the same package, and vendors must not call it.

‎docs/source/contributor-guide/s3-credential-provider-design.md‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -73,21 +73,22 @@ Comet's bridge does not maintain a TTL cache, schedule refresh, or broadcast cat
7373

7474
A Comet-side cache would have to either expose a tuning knob (TTL, max size, eviction policy) and grow over time, or be hardcoded and surprise vendors whose policies disagree. The bridge intentionally has neither and forwards every call.
7575

76-
## Scope-aware `object_store` registry
76+
## Location-scoped credentials on the Parquet path
7777

78-
The base SPI hands Comet one credential per `(bucket, path)`. Vendors whose sessions are inherently scoped (STS session A covers `s3://bucket/prefix-A/**` but not `s3://bucket/prefix-B/**`) hit a subtle failure mode against the pre-scope `object_store` cache: the cache key was `(scheme://bucket, config_hash, hdfs_backend)`, so the *first* prefix's session bound at construction time was reused for every later prefix under the same bucket, and any later prefix that fell outside that session's scope 403'd on read.
78+
`object_store::CredentialProvider::get_credential` receives no request path, so one `AmazonS3` store presents one credential, and the process-wide `object_store` cache holds one store per `(scheme://bucket, config_hash, hdfs_backend)`. A base provider therefore gets one credential per bucket, requested with the path of the first file Comet reads. A bucket whose policies differ by location (one for `warehouse/sales`, another for `warehouse/finance`) needs a store per location and something that picks among them for each request.
7979

80-
The opt-in `CometS3ScopedCredentialProvider` sub-interface + a correctness safety net on top of the cache resolves this without changing the base contract:
80+
`CometS3LocationScopedCredentialProvider` supplies that. `getPolicyLocations(bucket)` returns every location in the bucket that has its own policy, and `create_store` returns a `LocationScopedObjectStore` for the bucket instead of a plain store:
8181

82-
1. **Advisory scope hint on the read path.** When Comet detects that the configured provider implements the sub-interface, it calls `getPolicyLocationsFor(context)` immediately after resolving the SPI class and before looking the store up in the cache. The returned prefixes are attached to each cache entry as a `ScopeEntry { prefixes, store }`; the outer key stays `(scheme://bucket, config_hash, hdfs_backend)`, but the value becomes `Vec<ScopeEntry>` and the read path picks the entry whose covering prefix is the *longest* match for the requested path (ties break by insertion order). An empty prefix list is treated as "catchall" and matches with effective length 0, so any properly-scoped covering entry wins — this preserves the pre-existing single-entry-per-bucket semantics for base-interface providers while letting scoped and catchall entries coexist under one key.
82+
- It is cached and registered like any other S3 store, one per key, so later scans on the bucket share it and the cache and registry keep one store per identity. As for any store, scans that miss the cache at the same moment each build one, and the last one cached wins.
83+
- Each request is served by the store of the longest location covering its path, matched one segment at a time after percent-decoding, with the bucket root as an implicit location. Routing is per request, so a partition whose files span several locations reads each file with its own location's credential.
84+
- A location's store is an `AmazonS3` whose bridge is bound to the location itself, so the vendor sees one stable path per location. It is built on first use, usually inside an async read, from an `S3StoreTemplate` that resolved the region when the bucket's store was created, so building never blocks on the Tokio runtime.
85+
- The locations are a snapshot. A 403 is either a real denial or a location added or removed since the snapshot, so the store fetches the locations again and retries the request once if its path now routes elsewhere. Every attempt, successful or not, starts a new snapshot generation, so requests routed from the same generation share one attempt and a failed attempt fails them all instead of each calling the provider in turn. The retry budget is per request, with no state that outlives it.
8386

84-
2. **`RetryOn403ObjectStore` safety net.** Every scoped store is wrapped in a thin `ObjectStore` decorator that intercepts `Error::PermissionDenied` (403) exactly once, invokes a caller-supplied rebuild closure, and retries the same operation against the rebuilt store. On a second 403 the error propagates. Only 403 triggers the retry: `Unauthenticated` (401) is treated as a permanent credential-config error and passes through. The rebuild closure receives the `Path` that triggered the 403, constructs a fresh bridge bound to that path, re-fires the SPI (so the vendor's `getPolicyLocationsFor` reflects the *actual* failing request), and appends a new `ScopeEntry` to the outer cache alongside the pre-existing entries. Longest-prefix match on the lookup side then routes subsequent requests inside the new scope to the fresh store, while requests inside the pre-existing scope continue to reuse the original — disjoint scoped stores on the same bucket coexist across 403 recoveries.
87+
The locations come from asking for the bucket's whole list rather than which prefixes one session covers. Asking per session leaves Comet to discover the other scopes from 403s, which needs mutable per-store state and cannot route a partition that spans scopes. With the whole list up front, routing is a function of the path.
8588

86-
The wrapper's scope of interception is deliberately narrow: `put_opts`, `get_opts`, `get_ranges`, `list_with_delimiter`, `copy_opts`, `rename_opts`. Stream variants (`list`, `list_with_offset`, `delete_stream`) surface per-item errors as-is, since per-item retry would require materializing the stream and there is no observed failure mode outside `get_opts`/`get_ranges` on the parquet path. `put_multipart_opts` also skips retry — a partially-uploaded multipart cannot be transparently restarted. For `copy_opts` and `rename_opts` the read-side (source) path is what the rebuild is scoped against.
89+
This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the store caches no credentials, and every request still calls `getCredentialsForPath`. What it keeps is the provider's location list and a store for each location that has been read, so it grows with the vendor's policy list, not with the paths read. The list is only fetched again after a 403, so a change that causes none is not seen until the executor builds a new store.
8790

88-
**Why not fold scope prefixes into the outer cache key.** The obvious alternative shape — extend `ObjectStoreCacheKey` to `(bucket, config_hash, hdfs_backend, Vec<String>)` — was rejected because the same underlying store is still a valid answer for any *sub*-prefix of a scope. Keying on the exact prefix list would fragment the cache along every axis vendors report and defeat sharing on the common path. Threading the prefixes onto the entry, not the key, keeps the identity of the underlying store stable while letting the read path pick the correct scope per request.
89-
90-
**Why the safety net at all.** The SPI is authored per-vendor; overreporting is easy to write and hard to detect at plan time. The 403-retry wrapper turns a vendor-side scope bug into a correctness-preserving performance cost (one round-trip + one new cache entry) rather than a user-visible failure. It also handles the legitimate case where a scope was accurate at plan time and expired mid-scan.
91+
The dispatcher returns `null` for a provider that does not implement the interface without calling it, and Comet builds the same plain store as before. The Iceberg path does not use locations. Operations other than reads route by path without the 403 retry, since Comet only reads through these stores.
9192

9293
## Path-specific behavior
9394

‎docs/source/user-guide/latest/s3-credential-providers.md‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -201,32 +201,34 @@ public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) th
201201
}
202202
```
203203

204-
### Scope hints via `CometS3ScopedCredentialProvider`
204+
### Credentials per location
205205

206-
The base `CometS3CredentialProvider` gives Comet one credential per (bucket, path) call. Vendors whose credentials are inherently *scoped* — a single STS session covers `s3://bucket/prefix-A/**` but not `s3://bucket/prefix-B/**` — can implement the opt-in sub-interface `CometS3ScopedCredentialProvider` to let Comet keep multiple scoped stores side-by-side in a single bucket instead of caching a single session and 403-ing when it is asked to serve a path outside its scope.
206+
On the Parquet path, a `CometS3CredentialProvider` gets one credential per bucket: Comet requests it with the path of the first file it reads from the bucket and uses it for every file there. If your policies differ by location within a bucket, for example one policy for `warehouse/sales` and another for `warehouse/finance`, implement `CometS3LocationScopedCredentialProvider` and tell Comet where those locations are:
207207

208208
```java
209-
package org.apache.comet.cloud.s3;
209+
public final class MyLocationProvider implements CometS3LocationScopedCredentialProvider {
210+
@Override
211+
public List<String> getPolicyLocations(String bucket) throws Exception {
212+
return policyService.locationsWithPolicies(bucket); // ["warehouse/sales", "warehouse/finance"]
213+
}
210214

211-
public interface CometS3ScopedCredentialProvider extends CometS3CredentialProvider {
212-
/**
213-
* Advisory list of path prefixes (or absolute s3://... URIs) the credential
214-
* returned by {@link #getCredentialsForPath(CometS3CredentialContext)} for
215-
* the same {@code context} is valid against. An empty list means "unknown /
216-
* catchall"; Comet then falls back to the pre-existing single-entry-per-bucket
217-
* behavior.
218-
*/
219-
java.util.List<String> getPolicyLocationsFor(CometS3CredentialContext context);
215+
@Override
216+
public CometS3Credentials getCredentialsForPath(CometS3CredentialContext ctx) throws Exception {
217+
// ctx.getPath() is a returned location with a leading slash, or "/" for the bucket root.
218+
return sessionForLocation(ctx.getBucket(), ctx.getPath(), ctx.getMode());
219+
}
220220
}
221221
```
222222

223-
Comet uses the hint to key the native `object_store` registry as `(bucket, config_hash, backend, scope_prefixes)` instead of the base three-tuple, so two prefixes with disjoint scopes each get their own store. **The hint is advisory.** S3 itself is authoritative: if the vendor overreports a scope and S3 returns 403 anyway, the native cache transparently rebuilds the store under a widened (catchall) scope, retries the operation once, and continues. A second 403 propagates to Spark as a real error.
223+
Comet serves each request with the credential of the longest location that covers its path. A location covers a path when the path is the location itself or lies below it, compared one `/`-separated segment at a time, so `warehouse/sales` covers `warehouse/sales/part-0.parquet` but not `warehouse/sales_eu/part-0.parquet`. The bucket root covers every path that no returned location covers, and an empty list serves the whole bucket with the root's credential. Write locations the way `CometS3CredentialContext.getPath()` writes paths: percent-encoded, without the scheme or bucket name. A literal `%` must be written as `%25`; other characters may be left unencoded, and a leading or trailing `/` is optional. When several locations decode to the same path, Comet keeps the first.
224+
225+
Comet requests a location's credential by calling `getCredentialsForPath` with the location as the path, as you returned it but with a leading slash. Every request under a location shares that credential, so it must authorize every path the location is the longest match for, and your cache can key on the location. Locations apply to Comet's native Parquet reads only; Iceberg reads call `getCredentialsForPath` as they do for any provider.
224226

225-
**When you want this.** Overreporting has a runtime cost — a rebuild round-trip and a permanent widen of that bucket's cache — so only implement the sub-interface when your credential surface is genuinely per-prefix and stable at plan time. A vendor that always vends a bucket-wide session should stay on the base interface.
227+
**When Comet asks.** Comet calls `getPolicyLocations` when it creates the store for a bucket on an executor and keeps the answer for later reads of that bucket with the same S3 configuration. Reads that start at the same moment may each create a store and call it. If a read then fails with 403, Comet asks again, once for all the reads that failed on the same answer, and retries each read once if its path now falls under a different location, so a location added while a job runs is picked up. A location added or removed without causing a 403 is not seen until the executor creates a new store. Make `getPolicyLocations` thread-safe and independent of where it runs; it may be called on the driver or on executors.
226228

227-
**Interaction with `initialize`.** Scope hints are per-request, exactly like `getCredentialsForPath`. Comet does not cache them across requests; if your vendor's scope evolves during a job you may return a different list on each call.
229+
**Failures.** If `getPolicyLocations` throws or returns `null`, or returns a location that is `null` or invalid, the read fails. A location is invalid if, once decoded, it is not valid UTF-8 or has a segment that is empty, `.`, `..`, or contains a control character, so a URI such as `s3://bucket/a` is invalid too. Comet does not fall back to a broader credential.
228230

229-
**Backward compatibility.** Existing implementations that only implement `CometS3CredentialProvider` (not the `Scoped` sub-interface) behave exactly as before — Comet keys and caches per-bucket, and the 403-retry path stays inactive.
231+
**Backward compatibility.** Providers that implement only `CometS3CredentialProvider` are unaffected. Comet calls nothing new on them and keeps one credential per bucket, as before.
230232

231233
### Composing multiple credential backends
232234

0 commit comments

Comments
 (0)