Skip to content

Commit dbb0cd7

Browse files
committed
address more review comments.
1 parent ead968b commit dbb0cd7

5 files changed

Lines changed: 328 additions & 97 deletions

File tree

‎docs/source/contributor-guide/s3-credential-provider-design.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -77,13 +77,14 @@ A Comet-side cache would have to either expose a tuning knob (TTL, max size, evi
7777

7878
The "no Comet-side cache" rule above is about the vendor _bridge_. There is one Comet-owned credential provider that deliberately does cache: the EKS/IRSA web-identity provider in `native/core/src/cloud/s3/web_identity.rs`. It is not a vendor path -- there is no JVM SPI involved -- so the reasoning above does not apply.
7979

80-
It exists because the implicit IRSA credential path on the **Iceberg scan** mishandles STS throttling. `AssumeRoleWithWebIdentity` is resolved per reader thread; a concurrent startup burst throttles STS; opendal's default reqsign chain does not retry the throttle and falls through to the EKS node instance role, which lacks bucket access, turning a transient throttle into a hard `403` (this is the reported failure). The raw-Parquet path is intentionally left alone: its AWS SDK default chain already retries and stops on a provider error rather than downgrading, and — because `NativeConfig.extractObjectStoreOptions` forwards Hadoop's `core-default.xml` default for `fs.s3a.aws.credentials.provider` — a Parquet take-over gated on "no provider configured" could never engage anyway. So the take-over is Iceberg-only: when both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set and the catalog configures no explicit credentials, `iceberg_common.rs::build_s3_credential_loader` installs `WebIdentityCredentialProvider` (via a `CustomAwsCredentialLoader`) instead of returning `Ok(None)`. It:
80+
It exists because the implicit IRSA credential path on the **Iceberg scan** mishandles STS throttling. `AssumeRoleWithWebIdentity` is resolved per reader thread; a concurrent startup burst throttles STS; opendal's default reqsign chain does not retry the throttle and falls through to the EKS node instance role, which lacks bucket access, turning a transient throttle into a hard `403` (this is the reported failure). The raw-Parquet path is intentionally left alone: its AWS SDK default chain already retries and stops on a provider error rather than downgrading, and — because `NativeConfig.extractObjectStoreOptions` forwards Hadoop's `core-default.xml` default for `fs.s3a.aws.credentials.provider` — a Parquet take-over gated on "no provider configured" could never engage anyway. So the take-over is Iceberg-only, and covers both reads and writes (both build their `FileIO` through `load_file_io`): when both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set, a region is set, and the catalog configures no explicit credentials, `iceberg_common.rs::build_s3_credential_loader` installs `WebIdentityCredentialProvider` (via a `CustomAwsCredentialLoader`) instead of returning `Ok(None)`. It:
8181

8282
- builds its STS client from the AWS SDK's fully-resolved `SdkConfig` (`aws_config::defaults(...).load()`) with a raised `RetryConfig`, and calls `AssumeRoleWithWebIdentity` on it; because the client comes from the resolved config rather than a hand-assembled one, it honors region, `use_fips`, `use_dual_stack` and any profile/custom STS endpoint with no per-setting copying to keep in sync,
83+
- reports the credential's real STS expiry (not an early deadline). reqsign's signer refuses to sign within ~10s of the reported expiry and reloads within 120s of it, so an early deadline would open a signing dead zone before every refresh; instead our own cache margin (`min_ttl`, floored to 120s) refreshes at or before the signer's reload point, and a still-valid credential keeps being served if a refresh is throttled,
8384
- only ever calls `AssumeRoleWithWebIdentity` -- no credential chain, no IMDS/instance-role fallback -- so a throttle that outlasts the retries errors instead of downgrading, and
84-
- caches one credential per process keyed by identity `(role_arn, token_file, region)` plus the resolved retry/refresh settings (so a catalog's own knobs are honored regardless of init order), single-flights refreshes so a reader burst triggers one STS call, briefly remembers a failed refresh so a throttled burst also costs one call rather than one per reader, and jitters the refresh deadline per process so cluster-wide refreshes desync.
85+
- caches one credential per process keyed by identity `(role_arn, token_file, region)` plus the resolved settings (so a catalog's own knobs are honored regardless of init order) and single-flights refreshes, so a startup burst makes one STS call per executor rather than one per reader thread; a failed refresh is briefly remembered so a throttled burst also costs one call rather than one per reader.
8586

86-
Unlike the bridge, this provider owns the whole credential lifecycle (there is no vendor to delegate to), which is why caching lives here. Config keys are catalog properties under the `s3.` prefix (`s3.comet.credential.webIdentity.*`), the same route as `s3.comet.credential.provider.class`; see the user guide "EKS / IRSA" section. It stands aside for any credential source opendal's default chain ranks ahead of web-identity -- a bridge class, catalog static keys / `client.assume-role.arn`, static `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, or a configured profile (`AWS_PROFILE`, or a shared credentials / config file). Deferring on a present profile/config file also covers a profile-configured custom STS `endpoint_url`. So it only changes the otherwise-default behavior.
87+
Unlike the bridge, this provider owns the whole credential lifecycle (there is no vendor to delegate to), which is why caching lives here. Config keys are catalog properties under the `s3.` prefix (`s3.comet.credential.webIdentity.*`), the same route as `s3.comet.credential.provider.class`; see the user guide "EKS / IRSA" section. It stands aside for any credential source opendal's default chain ranks ahead of web-identity -- a bridge class, catalog static keys / `client.assume-role.arn`, static `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, or a configured profile (`AWS_PROFILE`, or a shared credentials / config file) -- and when no region is set (a web-identity STS client with no region fails opaquely, while the default chain has a global-endpoint fallback). Deferring on a present profile/config file also covers a profile-configured custom STS `endpoint_url`. So it only changes the otherwise-default behavior.
8788

8889
## Path-specific behavior
8990

‎docs/source/user-guide/latest/s3-credential-providers.md‎

Lines changed: 13 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -100,28 +100,27 @@ Without the config set, no credential-related log lines appear at startup; nativ
100100

101101
**Credentials silently going stale during long-running jobs.** When a vendor returns `expirationEpochMillis=0`, the bridge substitutes a 5-minute expiry before handing the credential to `opendal`, so `opendal`'s cache cannot hold a stale credential indefinitely. Returning a real expiry is preferred; the 5-minute fallback is a safety net, not a knob.
102102

103-
## EKS / IRSA: STS throttling protection (native Iceberg scan)
103+
## EKS / IRSA: STS throttling protection (native Iceberg reads and writes)
104104

105-
This is automatic; there is nothing to configure to get the protection, and it does not involve a bridge class. It applies to the **native Iceberg scan**. The raw-Parquet path is unaffected: it uses the AWS SDK default chain, which already retries and stops on a provider error rather than downgrading.
105+
This is automatic; there is nothing to configure to get the protection, and it does not involve a bridge class. It applies to native Iceberg **reads and writes** (both build their `FileIO` through the same path). The raw-Parquet path is unaffected: it uses the AWS SDK default chain, which already retries and stops on a provider error rather than downgrading.
106106

107107
On EKS with [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html), executors assume the application role by calling STS `AssumeRoleWithWebIdentity`. Under a large concurrent startup burst (many executors times many cores, all fetching credentials at once), STS can throttle that call. On the Iceberg path, opendal's default credential chain does not retry the throttle and falls through to the EKS node instance role, which usually lacks bucket access, so every native read then fails with a hard `403 AccessDenied` even though the throttle was transient.
108108

109-
When Comet detects IRSA (both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set) and the catalog configures no explicit credentials, the native Iceberg reader resolves web-identity credentials itself instead of using opendal's default chain. It builds the STS client from the AWS SDK's fully-resolved config, so region, `AWS_USE_FIPS_ENDPOINT` / `AWS_USE_DUALSTACK_ENDPOINT`, and any profile/custom STS endpoint are honored. The provider:
109+
When Comet detects IRSA (both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set), a region is set, and the catalog configures no explicit credentials, the native Iceberg reader resolves web-identity credentials itself instead of using opendal's default chain. It builds the STS client from the AWS SDK's fully-resolved config, so region, `AWS_USE_FIPS_ENDPOINT` / `AWS_USE_DUALSTACK_ENDPOINT`, and any profile/custom STS endpoint are honored. The provider:
110110

111-
- retries the throttled `AssumeRoleWithWebIdentity` call with exponential backoff and jitter,
112-
- never falls back to the node instance role -- a throttle that outlasts the retries surfaces as a retryable error instead of a wrong-identity `403`, and
113-
- caches one assumed-role credential per executor process, shared across all reader threads and scans, and refreshes it ahead of expiry with a per-process jitter so cluster-wide refreshes do not synchronize into another burst.
111+
- retries the throttled `AssumeRoleWithWebIdentity` call with the AWS SDK's exponential backoff and jitter (a throttle that outlasts the retries becomes an error that Spark's task retry then picks up),
112+
- never falls back to the node instance role, and
113+
- caches one assumed-role credential per executor process, shared across all reader threads and scans, so a startup burst makes one STS call per executor rather than one per thread. If a refresh is throttled while the current credential is still valid, it keeps serving that credential.
114114

115-
It stands aside whenever a higher-precedence credential source is configured -- a Comet bridge class or catalog static keys / `client.assume-role.arn`, static credentials in the environment (`AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`), or a configured profile (`AWS_PROFILE`, or a shared credentials / config file such as `~/.aws/credentials` or `~/.aws/config`). These all rank ahead of web-identity in opendal's default chain, so the take-over only changes the otherwise-default behavior and never switches away from an identity -- or a profile-configured STS endpoint -- you set explicitly. On an EKS/IRSA pod none of these are normally present, so the take-over still applies there.
115+
It stands aside whenever a higher-precedence credential source is configured -- a Comet bridge class or catalog static keys / `client.assume-role.arn`, static credentials in the environment (`AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`), or a configured profile (`AWS_PROFILE`, or a shared credentials / config file such as `~/.aws/credentials` or `~/.aws/config`) -- and when no region is set (`AWS_REGION` / `AWS_DEFAULT_REGION`). These all rank ahead of web-identity in opendal's default chain (or, for a missing region, need its global-endpoint fallback), so the take-over only changes the otherwise-default behavior and never switches away from an identity -- or a profile-configured STS endpoint -- you set explicitly. On an EKS/IRSA pod none of these apply, so the take-over still engages there.
116116

117117
Tuning is rarely needed. Set these as catalog properties under the `s3.` prefix (the same namespace as the credential-provider SPI key):
118118

119-
| Property (per Iceberg catalog) | Default | Meaning |
120-
| ------------------------------------------------------ | ------- | -------------------------------------------------------------- |
121-
| `s3.comet.credential.webIdentity.enabled` | `true` | Set `false` to opt out and use opendal's default chain. |
122-
| `s3.comet.credential.webIdentity.maxAttempts` | `5` | STS attempts before the assume-role call is treated as failed. |
123-
| `s3.comet.credential.webIdentity.minTtlSeconds` | `300` | Refresh this many seconds before expiry. |
124-
| `s3.comet.credential.webIdentity.refreshJitterSeconds` | `60` | Upper bound on the extra per-process refresh jitter. |
119+
| Property (per Iceberg catalog) | Default | Meaning |
120+
| ----------------------------------------------- | ------- | -------------------------------------------------------------- |
121+
| `s3.comet.credential.webIdentity.enabled` | `true` | Set `false` to opt out and use opendal's default chain. |
122+
| `s3.comet.credential.webIdentity.maxAttempts` | `5` | STS attempts before the assume-role call is treated as failed. |
123+
| `s3.comet.credential.webIdentity.minTtlSeconds` | `300` | Refresh this many seconds before expiry (floored to 120s). |
125124

126125
For example, to opt out of the take-over or raise the retry count for one catalog:
127126

@@ -130,7 +129,7 @@ spark.sql.catalog.<catalog>.s3.comet.credential.webIdentity.enabled=false
130129
spark.sql.catalog.<catalog>.s3.comet.credential.webIdentity.maxAttempts=8
131130
```
132131

133-
The `s3.` prefix is required — that is how a catalog property reaches the native reader (the same route the credential-provider SPI key uses); a key without it is dropped and has no effect.
132+
Use the `s3.` prefix. Comet forwards the unfiltered catalog property bag, so a bare, unprefixed key does technically reach the native reader, but the settings are looked up under `s3.` (matching the credential-provider SPI key), so only the prefixed spelling takes effect.
134133

135134
## Iceberg: explicit S3 region required
136135

‎native/Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎native/core/Cargo.toml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,9 +106,11 @@ assertables = "10"
106106
hex = "0.4.3"
107107
datafusion-functions-nested = { version = "55.1.0" }
108108
# Test-only helpers for a hand-rolled in-memory HTTP client that exercises the web-identity STS
109-
# retry path without a socket. Already in the dependency graph (transitive via aws-config).
109+
# retry path without a socket, plus reqsign's SigV4 signer for the signing round-trip test. Already
110+
# in the dependency graph (transitive via aws-config and iceberg-storage-opendal).
110111
aws-smithy-types = "1.6.3"
111112
http = "1.5.0"
113+
reqsign-aws-v4 = "3"
112114

113115
[features]
114116
backtrace = ["datafusion/backtrace"]

0 commit comments

Comments
 (0)