Skip to content

Commit ce455f3

Browse files
authored
feat: built-in S3 credential provider adapters for the native Parquet scan (apache#6023)
* feat: built-in S3 credential provider adapters for the native Parquet scan * more comments addressed * address review comments * format
1 parent 7649361 commit ce455f3

35 files changed

Lines changed: 2468 additions & 23 deletions

‎dev/ci/check-suites.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ def file_to_class_name(path: Path) -> str | None:
3737
"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite", # manual test suite (loads libhdfs, see #5023)
3838
"org.apache.comet.IcebergReadFromS3Suite", # manual test suite
3939
"org.apache.comet.cloud.s3.CometS3CredentialBridgeSuite", # manual test suite
40+
"org.apache.comet.cloud.s3.HadoopS3ACredentialProviderAdapterBridgeSuite", # manual test suite
4041
"org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite", # dedicated version matrix
4142
"org.apache.spark.sql.comet.CometPlanStabilitySuite", # abstract
4243
"org.apache.spark.sql.comet.ParquetDatetimeRebaseSuite", # abstract

‎docs/source/about/versioning_policy.md‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -257,10 +257,12 @@ says, and is covered by [Everything Else Is Internal](#everything-else-is-intern
257257
These classes are named as _values_ in Spark configuration properties. Users do not compile against
258258
them; they write the fully qualified name into a config string.
259259

260-
| Class | Named in | Purpose |
261-
| -------------------------------------- | ------------------------------------ | ----------------------------------------------------- |
262-
| `org.apache.spark.CometPlugin` | `spark.plugins` | Installs Comet. |
263-
| `org.apache.comet.ExtendedExplainInfo` | `spark.sql.extendedExplainProviders` | Adds Comet fallback explanations to `EXPLAIN` output. |
260+
| Class | Named in | Purpose |
261+
| -------------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------ |
262+
| `org.apache.spark.CometPlugin` | `spark.plugins` | Installs Comet. |
263+
| `org.apache.comet.ExtendedExplainInfo` | `spark.sql.extendedExplainProviders` | Adds Comet fallback explanations to `EXPLAIN` output. |
264+
| `org.apache.comet.cloud.s3.HadoopS3ACredentialProviderAdapter` | `fs.s3a.comet.credential.provider.class` | Built-in adapter delegating to Hadoop S3A's provider construction. |
265+
| `org.apache.comet.cloud.s3.AwsSdkCredentialProviderAdapter` | `fs.s3a.comet.credential.provider.class` | Built-in adapter wrapping a raw AWS SDK provider. |
264266

265267
Renaming or removing one of these class names breaks user configuration in exactly the way renaming
266268
a `spark.comet.*` key does, so it is treated on the same terms: a deprecation cycle, then removal in
@@ -281,6 +283,14 @@ The SPI consists of:
281283
- `CometS3Credentials`, the value a provider returns.
282284
- `CometS3CredentialContext` and `CometS3AccessMode`, describing the request being served.
283285

286+
Comet also ships two built-in implementations, `HadoopS3ACredentialProviderAdapter` and
287+
`AwsSdkCredentialProviderAdapter`. Users do not compile against them; they name them in
288+
`fs.s3a.comet.credential.provider.class`, so they are covered by the config-referenced class-name
289+
table above (name pinned, internal structure not guaranteed) rather than by the SPI's binary
290+
compatibility contract. See the
291+
[S3 Credential Providers](../user-guide/latest/s3-credential-providers.md) guide for how to enable
292+
them.
293+
284294
Additive changes are allowed in a minor release, for example a new accessor on
285295
`CometS3CredentialContext`, because a vendor jar compiled against an earlier `1.x` continues to
286296
load and run. Any change that would break such a jar, including adding an abstract method to

‎docs/source/contributor-guide/s3-credential-provider-design.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,19 @@ The full unfiltered FileIO property bag crosses JNI as `catalog_properties`. The
116116

117117
This is consistent with [Why no Comet-side cache](#why-no-comet-side-cache): the cache holds the `FileIO` and its bridge, not credentials. `provide_credential` still reaches the vendor whenever `opendal`'s cached credential expires.
118118

119+
## Property-bag handling on the Parquet path
120+
121+
The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so stripping the static keys would let a chain such as `SimpleAWSCredentialsProvider,customProvider` silently resolve through a different provider than Spark, reading data as a different principal. These keys already cross JNI for the non-adapter native path (`build_credential_provider` reads them), so this matches existing behavior rather than widening exposure. Because `NativeConfig.extractObjectStoreOptions` only collects `fs.s3a.*` (plus the `fs.comet.*` scheme keys), config a provider reads that is not under `fs.s3a.*` -- notably `hadoop.security.credential.provider.path` -- never crosses JNI. `AdapterSupport.toConfiguration` therefore seeds the Hadoop `Configuration` from the executor's own Spark-derived Hadoop conf (`spark.hadoop.*`) and overlays the forwarded `fs.s3a.*` on top, so those keys are present.
122+
123+
## Built-in adapters
124+
125+
Comet ships two reference SPI implementations under `org.apache.comet.cloud.s3`, so standard provider classes that the native Rust list does not match work with a config change instead of bespoke code:
126+
127+
- `HadoopS3ACredentialProviderAdapter` delegates to Hadoop S3A's own provider construction (`S3AUtils.createAWSCredentialProviderSet` on Hadoop 3.3.x, `CredentialProviderListFactory.createAWSCredentialProviderList` on 3.4+).
128+
- `AwsSdkCredentialProviderAdapter` wraps a raw AWS SDK provider named in `fs.s3a.comet.credential.adapter.class`.
129+
130+
Each has a spark-3.x (SDK v1) and a spark-4.x (SDK v2) body under the same FQCN, selected by the `shims.majorVerSrc` source set, so each Comet build compiles against exactly the one AWS SDK its Hadoop line ships. The SDK and `hadoop-aws` are `provided` scope only (see the `hadoop-aws.version` property in the root `pom.xml`), so Comet does not bundle a second copy.
131+
119132
## Returns or throws, not a fall-through value
120133

121134
The SPI returns a `CometS3Credentials` or throws. There is no sentinel "I do not know" return. Vendors that are only authoritative for some paths resolve the default AWS chain themselves for the rest and return the result. This matches the contract on every other AWS credential SPI in the JVM ecosystem (AWS SDK v1/v2, Hadoop S3A, Iceberg `VendedCredentialsProvider`).

‎docs/source/user-guide/latest/s3-credential-providers.md‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,47 @@ You probably do, if any of these are true:
3636
- You have a custom Iceberg `client.factory` that injects a configured S3 client.
3737
- Spark queries against your S3 paths work, but the same queries with Comet enabled fail with 403.
3838

39+
## Built-in adapters
40+
41+
If a native Parquet scan fails with `Unsupported credential provider: <class>` (for example `com.amazonaws.auth.DefaultAWSCredentialsProviderChain`), the class you named in `fs.s3a.aws.credentials.provider` is one that plain Spark/Hadoop accepts but Comet's native reader does not reimplement. Comet ships two built-in `CometS3CredentialProvider` adapters that fix this with a one-line config change; you leave your existing `fs.s3a.aws.credentials.provider` untouched.
42+
43+
These adapters cover the Parquet native scan path only. Enabling one is opt-in: naming it is what activates it. Note the native side forwards the `fs.s3a.*` config to `initialize()` for _any_ provider class named on the Parquet path, not just these two adapters: a vendor `CometS3CredentialProvider` that received an empty map in Comet 1.0 now receives the `fs.s3a.*` subset (including static keys), and one cached instance per distinct `fs.s3a.*` config rather than one per bucket. This is additive, but a provider that logs the map or treats an empty map as "the Parquet path" should be aware of it.
44+
45+
The adapters and the AWS SDK are loaded through the class loader that loaded Comet, so `hadoop-aws` and the matching AWS SDK must be visible from there — put them on the same classpath as Comet (`spark.executor.extraClassPath` / `spark.driver.extraClassPath`, or `$SPARK_HOME/jars`), not only via `--packages`. If they are only on the user-jar loader, credential resolution fails at planning with `NoClassDefFoundError` before the adapter can report anything useful.
46+
47+
### `HadoopS3ACredentialProviderAdapter` (recommended)
48+
49+
Delegates to Hadoop S3A's own provider construction, so it accepts everything the `fs.s3a.aws.credentials.provider` chain accepts (the default chain, web-identity, assumed-role, per-bucket config). This is the general answer for the failure above. It does **not** cover `fs.s3a.custom.signers` — the native reader signs SigV4 itself with whatever the chain returns and never invokes a custom signer, so a signer's identity would not be applied; a custom-signer setup needs a vendor bridge (see "Do I need this?"). And it **refuses** `fs.s3a.delegation.token.binding`: Spark would use the delegation-token provider and bypass the chain, so rather than resolve a different identity the adapter fails with a message naming that key.
50+
51+
```
52+
spark.hadoop.fs.s3a.comet.credential.provider.class=org.apache.comet.cloud.s3.HadoopS3ACredentialProviderAdapter
53+
# leave your existing config as-is, for example:
54+
spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
55+
```
56+
57+
It needs no extra config: it reads the standard `fs.s3a.aws.credentials.provider` (and the per-bucket `fs.s3a.bucket.<bucket>.aws.credentials.provider`) itself, and Comet forwards the full `fs.s3a.*` config to it, so a provider chain (including static keys and assumed-role) resolves the same way it would under Spark.
58+
59+
Anonymous access is the one exception. A bucket whose chain resolves to `AnonymousAWSCredentialsProvider` (common for public datasets) is not supported: the SPI has no way to express "anonymous", so the adapter fails with an error naming the bucket rather than reading it unsigned. To read such a bucket, opt it out of the adapter with an empty per-bucket class so the native reader accesses it directly:
60+
61+
```
62+
spark.hadoop.fs.s3a.bucket.<public-bucket>.comet.credential.provider.class=
63+
```
64+
65+
### `AwsSdkCredentialProviderAdapter`
66+
67+
Wraps a single raw AWS SDK credential-provider class that is not registered through S3A. Name the delegate in a separate key:
68+
69+
```
70+
spark.hadoop.fs.s3a.comet.credential.provider.class=org.apache.comet.cloud.s3.AwsSdkCredentialProviderAdapter
71+
spark.hadoop.fs.s3a.comet.credential.adapter.class=<FQCN of your credential provider>
72+
# per-bucket variant:
73+
spark.hadoop.fs.s3a.bucket.<bucket>.comet.credential.adapter.class=<FQCN>
74+
```
75+
76+
### Which one, and which Spark version
77+
78+
Use `HadoopS3ACredentialProviderAdapter` unless you have a plain SDK provider not wired through S3A. Both class names are the same on every Comet build; each build automatically uses the AWS SDK its Hadoop line ships (v1 on the Spark 3.4/3.5 builds, v2 on 4.0+), so you configure one name and get the right implementation.
79+
3980
## Enabling a bridge
4081

4182
A bridge is activated by naming the vendor's class in a Spark config. Putting a JAR on the classpath alone has no effect; the config key must be set.
@@ -88,6 +129,8 @@ Without the config set, no credential-related log lines appear at startup; nativ
88129

89130
## Troubleshooting
90131

132+
**`Generic S3 error: Unsupported credential provider: <class>`** (native Parquet scan). The class in `fs.s3a.aws.credentials.provider` is one Hadoop S3A accepts but Comet's native reader does not reimplement. Name `HadoopS3ACredentialProviderAdapter` as the Comet provider class (see [Built-in adapters](#built-in-adapters)) and leave your existing config alone.
133+
91134
**`CometS3CredentialProvider class not found: <name>`**. The class named in the config is not on the executor classpath. Re-check `--jars` / `spark.jars`. On YARN or Kubernetes, confirm the JAR actually reached the executor and not only the driver.
92135

93136
**`<class> does not implement org.apache.comet.cloud.s3.CometS3CredentialProvider`**. The configured class exists but does not implement the SPI. Double-check the FQCN against the vendor's documentation.
@@ -136,7 +179,7 @@ The class must have a public no-arg constructor. `getCredentialsForPath` may be
136179

137180
Comet keys provider instances by `(FQCN, dispatchKey, catalogProperties)`. The dispatch key is the Spark V2 catalog name on the Iceberg path and the S3 bucket name on the Parquet path. The first time a given key is seen on an executor, Comet reflects the class, calls `initialize(Map)` exactly once, and caches the instance for the JVM lifetime. Two catalogs sharing one provider FQCN therefore get isolated instances with their own `initialize` maps. Including `catalogProperties` in the key matters in multi-tenant JVMs (Spark Connect, Thrift Server, `SparkSession.newSession()`) where two sessions can otherwise collide on the same `(FQCN, dispatchKey)` and have the second session silently use the first session's credentials.
138181

139-
`initialize` should be cheap and non-blocking. Defer real credential fetches (REST round-trips, STS calls) to the first `getCredentialsForPath` invocation. On the Iceberg path the supplied `catalogProperties` carries the unfiltered FileIO bag, including REST-vended fields like `credentials.uri`, OAuth tokens, and any vendor-custom keys you set on the catalog config. The map may contain secrets, so do not log it.
182+
`initialize` should be cheap and non-blocking. Defer real credential fetches (REST round-trips, STS calls) to the first `getCredentialsForPath` invocation. On the Iceberg path the supplied `catalogProperties` carries the unfiltered FileIO bag, including REST-vended fields like `credentials.uri`, OAuth tokens, and any vendor-custom keys you set on the catalog config. On the Parquet path it carries the `fs.s3a.*` config subset (including static keys such as `fs.s3a.access.key`); in Comet 1.0 this map was empty on the Parquet path. The map may contain secrets, so do not log it.
140183

141184
`close()` is invoked from a JVM shutdown hook installed by the dispatcher. The default no-op is fine for stateless providers. Override it to release HTTP clients, scheduled-refresh executors, or STS connection pools. Shutdown hooks are best-effort: a `SIGKILL` or abrupt JVM termination skips them, so do not depend on `close()` for correctness.
142185

0 commit comments

Comments
 (0)