You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dbb0cd7
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/s3-credential-provider-design.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -77,13 +77,14 @@ A Comet-side cache would have to either expose a tuning knob (TTL, max size, evi
77
77
78
78
The "no Comet-side cache" rule above is about the vendor _bridge_. There is one Comet-owned credential provider that deliberately does cache: the EKS/IRSA web-identity provider in `native/core/src/cloud/s3/web_identity.rs`. It is not a vendor path -- there is no JVM SPI involved -- so the reasoning above does not apply.
79
79
80
-
It exists because the implicit IRSA credential path on the **Iceberg scan** mishandles STS throttling. `AssumeRoleWithWebIdentity` is resolved per reader thread; a concurrent startup burst throttles STS; opendal's default reqsign chain does not retry the throttle and falls through to the EKS node instance role, which lacks bucket access, turning a transient throttle into a hard `403` (this is the reported failure). The raw-Parquet path is intentionally left alone: its AWS SDK default chain already retries and stops on a provider error rather than downgrading, and — because `NativeConfig.extractObjectStoreOptions` forwards Hadoop's `core-default.xml` default for `fs.s3a.aws.credentials.provider` — a Parquet take-over gated on "no provider configured" could never engage anyway. So the take-over is Iceberg-only: when both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set and the catalog configures no explicit credentials, `iceberg_common.rs::build_s3_credential_loader` installs `WebIdentityCredentialProvider` (via a `CustomAwsCredentialLoader`) instead of returning `Ok(None)`. It:
80
+
It exists because the implicit IRSA credential path on the **Iceberg scan** mishandles STS throttling. `AssumeRoleWithWebIdentity` is resolved per reader thread; a concurrent startup burst throttles STS; opendal's default reqsign chain does not retry the throttle and falls through to the EKS node instance role, which lacks bucket access, turning a transient throttle into a hard `403` (this is the reported failure). The raw-Parquet path is intentionally left alone: its AWS SDK default chain already retries and stops on a provider error rather than downgrading, and — because `NativeConfig.extractObjectStoreOptions` forwards Hadoop's `core-default.xml` default for `fs.s3a.aws.credentials.provider` — a Parquet take-over gated on "no provider configured" could never engage anyway. So the take-over is Iceberg-only, and covers both reads and writes (both build their `FileIO` through `load_file_io`): when both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set, a region is set, and the catalog configures no explicit credentials, `iceberg_common.rs::build_s3_credential_loader` installs `WebIdentityCredentialProvider` (via a `CustomAwsCredentialLoader`) instead of returning `Ok(None)`. It:
81
81
82
82
- builds its STS client from the AWS SDK's fully-resolved `SdkConfig` (`aws_config::defaults(...).load()`) with a raised `RetryConfig`, and calls `AssumeRoleWithWebIdentity` on it; because the client comes from the resolved config rather than a hand-assembled one, it honors region, `use_fips`, `use_dual_stack` and any profile/custom STS endpoint with no per-setting copying to keep in sync,
83
+
- reports the credential's real STS expiry (not an early deadline). reqsign's signer refuses to sign within ~10s of the reported expiry and reloads within 120s of it, so an early deadline would open a signing dead zone before every refresh; instead our own cache margin (`min_ttl`, floored to 120s) refreshes at or before the signer's reload point, and a still-valid credential keeps being served if a refresh is throttled,
83
84
- only ever calls `AssumeRoleWithWebIdentity` -- no credential chain, no IMDS/instance-role fallback -- so a throttle that outlasts the retries errors instead of downgrading, and
84
-
- caches one credential per process keyed by identity `(role_arn, token_file, region)` plus the resolved retry/refresh settings (so a catalog's own knobs are honored regardless of init order), single-flights refreshes so a reader burst triggers one STS call, briefly remembers a failed refresh so a throttled burst also costs one call rather than one per reader, and jitters the refresh deadline per process so cluster-wide refreshes desync.
85
+
- caches one credential per process keyed by identity `(role_arn, token_file, region)` plus the resolved settings (so a catalog's own knobs are honored regardless of init order) and single-flights refreshes, so a startup burst makes one STS call per executor rather than one per reader thread; a failed refresh is briefly remembered so a throttled burst also costs one call rather than one per reader.
85
86
86
-
Unlike the bridge, this provider owns the whole credential lifecycle (there is no vendor to delegate to), which is why caching lives here. Config keys are catalog properties under the `s3.` prefix (`s3.comet.credential.webIdentity.*`), the same route as `s3.comet.credential.provider.class`; see the user guide "EKS / IRSA" section. It stands aside for any credential source opendal's default chain ranks ahead of web-identity -- a bridge class, catalog static keys / `client.assume-role.arn`, static `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, or a configured profile (`AWS_PROFILE`, or a shared credentials / config file). Deferring on a present profile/config file also covers a profile-configured custom STS `endpoint_url`. So it only changes the otherwise-default behavior.
87
+
Unlike the bridge, this provider owns the whole credential lifecycle (there is no vendor to delegate to), which is why caching lives here. Config keys are catalog properties under the `s3.` prefix (`s3.comet.credential.webIdentity.*`), the same route as `s3.comet.credential.provider.class`; see the user guide "EKS / IRSA" section. It stands aside for any credential source opendal's default chain ranks ahead of web-identity -- a bridge class, catalog static keys / `client.assume-role.arn`, static `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, or a configured profile (`AWS_PROFILE`, or a shared credentials / config file) -- and when no region is set (a web-identity STS client with no region fails opaquely, while the default chain has a global-endpoint fallback). Deferring on a present profile/config file also covers a profile-configured custom STS `endpoint_url`. So it only changes the otherwise-default behavior.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/s3-credential-providers.md
+13-14Lines changed: 13 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -100,28 +100,27 @@ Without the config set, no credential-related log lines appear at startup; nativ
100
100
101
101
**Credentials silently going stale during long-running jobs.** When a vendor returns `expirationEpochMillis=0`, the bridge substitutes a 5-minute expiry before handing the credential to `opendal`, so `opendal`'s cache cannot hold a stale credential indefinitely. Returning a real expiry is preferred; the 5-minute fallback is a safety net, not a knob.
This is automatic; there is nothing to configure to get the protection, and it does not involve a bridge class. It applies to the **native Iceberg scan**. The raw-Parquet path is unaffected: it uses the AWS SDK default chain, which already retries and stops on a provider error rather than downgrading.
105
+
This is automatic; there is nothing to configure to get the protection, and it does not involve a bridge class. It applies to native Iceberg **reads and writes** (both build their `FileIO` through the same path). The raw-Parquet path is unaffected: it uses the AWS SDK default chain, which already retries and stops on a provider error rather than downgrading.
106
106
107
107
On EKS with [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html), executors assume the application role by calling STS `AssumeRoleWithWebIdentity`. Under a large concurrent startup burst (many executors times many cores, all fetching credentials at once), STS can throttle that call. On the Iceberg path, opendal's default credential chain does not retry the throttle and falls through to the EKS node instance role, which usually lacks bucket access, so every native read then fails with a hard `403 AccessDenied` even though the throttle was transient.
108
108
109
-
When Comet detects IRSA (both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set) and the catalog configures no explicit credentials, the native Iceberg reader resolves web-identity credentials itself instead of using opendal's default chain. It builds the STS client from the AWS SDK's fully-resolved config, so region, `AWS_USE_FIPS_ENDPOINT` / `AWS_USE_DUALSTACK_ENDPOINT`, and any profile/custom STS endpoint are honored. The provider:
109
+
When Comet detects IRSA (both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set), a region is set, and the catalog configures no explicit credentials, the native Iceberg reader resolves web-identity credentials itself instead of using opendal's default chain. It builds the STS client from the AWS SDK's fully-resolved config, so region, `AWS_USE_FIPS_ENDPOINT` / `AWS_USE_DUALSTACK_ENDPOINT`, and any profile/custom STS endpoint are honored. The provider:
110
110
111
-
- retries the throttled `AssumeRoleWithWebIdentity` call with exponential backoff and jitter,
112
-
- never falls back to the node instance role -- a throttle that outlasts the retries surfaces as a retryable error instead of a wrong-identity `403`, and
113
-
- caches one assumed-role credential per executor process, shared across all reader threads and scans, and refreshes it ahead of expiry with a per-process jitter so cluster-wide refreshes do not synchronize into another burst.
111
+
- retries the throttled `AssumeRoleWithWebIdentity` call with the AWS SDK's exponential backoff and jitter (a throttle that outlasts the retries becomes an error that Spark's task retry then picks up),
112
+
- never falls back to the node instance role, and
113
+
- caches one assumed-role credential per executor process, shared across all reader threads and scans, so a startup burst makes one STS call per executor rather than one per thread. If a refresh is throttled while the current credential is still valid, it keeps serving that credential.
114
114
115
-
It stands aside whenever a higher-precedence credential source is configured -- a Comet bridge class or catalog static keys / `client.assume-role.arn`, static credentials in the environment (`AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`), or a configured profile (`AWS_PROFILE`, or a shared credentials / config file such as `~/.aws/credentials` or `~/.aws/config`). These all rank ahead of web-identity in opendal's default chain, so the take-over only changes the otherwise-default behavior and never switches away from an identity -- or a profile-configured STS endpoint -- you set explicitly. On an EKS/IRSA pod none of these are normally present, so the take-over still applies there.
115
+
It stands aside whenever a higher-precedence credential source is configured -- a Comet bridge class or catalog static keys / `client.assume-role.arn`, static credentials in the environment (`AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`), or a configured profile (`AWS_PROFILE`, or a shared credentials / config file such as `~/.aws/credentials` or `~/.aws/config`) -- and when no region is set (`AWS_REGION` / `AWS_DEFAULT_REGION`). These all rank ahead of web-identity in opendal's default chain (or, for a missing region, need its global-endpoint fallback), so the take-over only changes the otherwise-default behavior and never switches away from an identity -- or a profile-configured STS endpoint -- you set explicitly. On an EKS/IRSA pod none of these apply, so the take-over still engages there.
116
116
117
117
Tuning is rarely needed. Set these as catalog properties under the `s3.` prefix (the same namespace as the credential-provider SPI key):
118
118
119
-
| Property (per Iceberg catalog) | Default | Meaning |
The `s3.` prefix is required — that is how a catalog property reaches the native reader (the same route the credential-provider SPI key uses); a key without it is dropped and has no effect.
132
+
Use the `s3.` prefix. Comet forwards the unfiltered catalog property bag, so a bare, unprefixed key does technically reach the native reader, but the settings are looked up under `s3.` (matching the credential-provider SPI key), so only the prefixed spelling takes effect.
0 commit comments