You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2b044b4
Browse filesBrowse the repository at this point in the historyBrowse files
fix: validate every Hadoop Azure auth mechanism and borrow only the token file for a named identity
Reject Hadoop configuration the native scan cannot honour instead of letting
object_store fall through to the Azure CLI or managed identity: a partial
OAuth principal, an explicit auth.type the translated keys do not satisfy, a
provider class other than the MSI, workload identity or client credentials
providers (matched by exact simple name), the untranslated SAS provider,
key provider, refresh token and user name or password keys, and blank values.
Translate fs.azure.sas.fixed.token below the container-scoped SAS key, take
the account from the host lowercased, and when Hadoop names only an identity
borrow AZURE_FEDERATED_TOKEN_FILE alone, erroring when no token file exists.
Every error carries one prefix and names the key, never a credential value.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/datasources.md
+39-19Lines changed: 39 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -352,43 +352,63 @@ Comet's Parquet scan reads Azure Data Lake Storage Gen2 (ADLS Gen2) through the
352
352
[`object_store` crate](https://crates.io/crates/object_store), using the `abfs` and `abfss` URL schemes.
353
353
As with S3, standard Hadoop ABFS configurations (the `fs.azure.*` keys you already set in `core-site.xml` or via `spark.hadoop.*`) are translated into the `object_store` crate's format, so existing configurations continue to work.
354
354
355
-
URLs use the same shape Spark and Hadoop emit: `abfss://<container>@<account>.dfs.core.windows.net/<path>`. The account is taken from the host and the container from the URL user-info. The `wasb`, `wasbs`, `az`, `azure`, and `adl` schemes are not supported by the native scan.
355
+
URLs use the same shape Spark and Hadoop emit: `abfss://<container>@<account>.dfs.core.windows.net/<path>`. The account is taken from the host and lowercased before any key lookup, so `MyAcct.dfs.core.windows.net` resolves `fs.azure.account.key.myacct.dfs.core.windows.net`; the container is taken from the URL user-info as written, which matches Hadoop, and Azure requires container names to be lowercase anyway. The `wasb`, `wasbs`, `az`, `azure`, and `adl` schemes are not supported by the native scan.
356
356
357
357
### Root CA Certificates
358
358
359
359
Azure scans discover Root CA Certificates the same way S3 scans do. See [Root CA Certificates](#root-ca-certificates) above. The Rust-based scan uses system Root CA Certificates rather than the Java Trust Store.
360
360
361
361
### Supported Authentication
362
362
363
-
The Hadoop `fs.azure.*` configuration is authoritative for authentication, matching Hadoop's own ABFS driver, which reads no environment variables. When the Hadoop keys configure a complete auth mechanism (a shared account key, a SAS token, a client secret, a federated token file, or an MSI endpoint), or `fs.azure.account.oauth.provider.type` names `MsiTokenProvider`, that mechanism alone determines the identity and no `AZURE_*` environment variable is consulted: not credentials such as `AZURE_STORAGE_TOKEN`, `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_FEDERATED_TOKEN_FILE`, and not transport settings such as `AZURE_ALLOW_HTTP`, `AZURE_PROXY_URL`, or `AZURE_STORAGE_ENDPOINT`, so the configured identity cannot be outranked, redirected, or intercepted by the environment. A partial Hadoop mechanism, such as `fs.azure.account.oauth2.token.file` without a client id and tenant, is not completed from the environment either; Hadoop requires the client id and tenant as well, and `object_store` then falls through its credential chain to the node's managed identity, so configure the full set.
363
+
The Hadoop `fs.azure.*` configuration is authoritative for authentication, matching Hadoop's own ABFS driver, which reads no environment variables. When the Hadoop keys configure a complete auth mechanism (a shared account key, a SAS token, a client secret, a federated token file, or an MSI endpoint), or `fs.azure.account.oauth.provider.type` names `MsiTokenProvider`, that mechanism alone determines the identity and no `AZURE_*` environment variable is consulted: not credentials such as `AZURE_STORAGE_TOKEN`, `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_FEDERATED_TOKEN_FILE`, and not transport settings such as `AZURE_ALLOW_HTTP`, `AZURE_PROXY_URL`, or `AZURE_STORAGE_ENDPOINT`, so the configured identity cannot be outranked, redirected, or intercepted by the environment. A partial Hadoop mechanism, such as `fs.azure.account.oauth2.token.file` or `fs.azure.account.oauth2.client.secret` without a client id and tenant, is not completed from the environment either: the scan fails with an error naming the missing keys, because Hadoop requires the client id and tenant as well and `object_store` would otherwise fall through its credential chain to the node's managed identity. A blank SAS token or credential value (`fs.azure.account.key`, `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.client.secret`, `fs.azure.account.oauth2.msi.tenant`, `fs.azure.account.oauth2.token.file`, and the other translated keys) is an error for the same reason, naming the exact key: `object_store` would send unsigned requests or fall through its chain.
364
364
365
-
When the Hadoop keys configure no auth mechanism, or only name the identity via `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.msi.tenant`, or `fs.azure.account.oauth2.msi.authority`, the `AZURE_*` environment variables (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_AUTHORITY_HOST`, `AZURE_STORAGE_*`) are honored and the Hadoop keys are applied on top. This is what makes AKS Workload Identity work in a stock pod with no extra configuration. Transport settings from the environment apply only in this case, alongside the environment credentials.
365
+
When the Hadoop keys configure no auth mechanism and name no identity, the `AZURE_*` environment variables (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_AUTHORITY_HOST`, `AZURE_STORAGE_*`) are honored. This is what makes AKS Workload Identity work in a stock pod with no extra configuration. Transport settings from the environment apply only in this case, alongside the environment credentials. The same rule covers `AZURE_USE_AZURE_CLI`, which makes `object_store` take its token from a logged-in Azure CLI, and `AZURE_SKIP_SIGNATURE`, which makes it send unsigned requests for public containers: both are honored only when Hadoop supplies neither a mechanism nor an identity, like every other `AZURE_*` variable.
366
+
367
+
When the Hadoop keys only name the identity, via `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.msi.tenant`, or `fs.azure.account.oauth2.msi.authority`, or via `fs.azure.account.oauth.provider.type` ending in `WorkloadIdentityTokenProvider` together with the client id and tenant, only `AZURE_FEDERATED_TOKEN_FILE` is read from the environment, and only when `fs.azure.account.oauth2.token.file` is not set. No other `AZURE_*` variable is consulted, so an ambient account key or bearer token cannot outrank the named principal. A named principal with no token file in either place is an error naming both sources, and a borrowed token file still needs the client id and tenant from the Hadoop keys.
| OAuth2 client credentials |`fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.client.secret`, `fs.azure.account.oauth2.client.endpoint` (tenant id is extracted from the endpoint URL) |
| SAS token |`fs.azure.sas.<container>.<account>`|
375
+
| SAS token |`fs.azure.sas.<container>.<account>`, `fs.azure.sas.fixed.token`|
376
+
377
+
This authority extends only to the mechanisms Comet recognises. An explicit `fs.azure.account.auth.type` is Hadoop choosing a mechanism, so it is validated against the translated keys rather than ignored, and Hadoop ABFS mechanisms that have no native counterpart are neither translated nor allowed to fall through to the environment or to managed identity. In each of these cases no `AZURE_*` variable is consulted, and when the translated keys do not satisfy the mechanism the scan fails with an error naming the key (and, for a class name, the value) instead:
|`fs.azure.account.auth.type` = `SharedKey`| Accepted only with `fs.azure.account.key`|
382
+
|`fs.azure.account.auth.type` = `OAuth`| Accepted with `fs.azure.account.oauth.provider.type` ending in `MsiTokenProvider`, `WorkloadIdentityTokenProvider` (with the client id and tenant, and a token file from `fs.azure.account.oauth2.token.file` or `AZURE_FEDERATED_TOKEN_FILE`, the only environment variable read in that case), or `ClientCredsTokenProvider` (with the client secret), or with no provider class and a client secret or token file; any other provider class is rejected, naming the class |
383
+
|`fs.azure.account.auth.type` = `Custom`| Loads a custom token provider class, which the native scan cannot run |
384
+
|`fs.azure.account.auth.type` = `SAS`| Rejected unless `fs.azure.sas.fixed.token` (or a container-scoped SAS token) is also set |
385
+
|`fs.azure.account.auth.type` = anything else | Rejected, naming the value; the native scan supports `SharedKey`, `OAuth` and `SAS`|
386
+
|`fs.azure.sas.token.provider.type`| Loads a SAS token provider class |
387
+
|`fs.azure.account.keyprovider`| Loads an account key provider class |
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is read only to decide whether Hadoop configures an auth mechanism. The storage account and container always come from the URL.
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is validated whenever it is set, with or without `fs.azure.account.auth.type`: a class ending in `MsiTokenProvider` stands alone, `WorkloadIdentityTokenProvider` needs the client id and tenant, `ClientCredsTokenProvider` needs the client secret principal, any other class is an error naming the key and the class, and a blank value is an error naming the key. A class matches only by its exact simple name, bare or as the last segment of a qualified name. It, `fs.azure.account.auth.type`, and the unsupported mechanism keys listed under [Supported Authentication](#supported-authentication) are read only to decide whether Hadoop configures an auth mechanism, and whether it is one the native scan can build. The storage account and container always come from the URL.
392
412
393
413
### Tenant id resolution
394
414
@@ -408,7 +428,7 @@ Account-scoped keys take precedence over global ones, mirroring Hadoop ABFS's ow
408
428
3.`<key>.<account>`
409
429
4.`<key>` (unscoped / global)
410
430
411
-
The SAS namespace follows the same shape with the container inlined into the key name: `fs.azure.sas.<container>.<account>.dfs.core.windows.net`, then `fs.azure.sas.<container>.<account>.blob.core.windows.net`, then `fs.azure.sas.<container>.<account>`.
431
+
The SAS namespace follows the same shape with the container inlined into the key name: `fs.azure.sas.<container>.<account>.dfs.core.windows.net`, then `fs.azure.sas.<container>.<account>.blob.core.windows.net`, then `fs.azure.sas.<container>.<account>`. When none of those is set, the account-level `fs.azure.sas.fixed.token` is probed in the account-scoped order above, so a container-scoped token always wins over a fixed token. A SAS token that is empty or only whitespace is rejected rather than treated as unset, and a blank container-scoped key is rejected even when a valid `fs.azure.sas.fixed.token` is set: the fixed token is never used as a fallback for it.
412
432
413
433
A Hadoop value always wins over an `AZURE_*` environment variable for the same key. When the Hadoop keys configure an auth mechanism, the environment's credential variables are not consulted at all (see [Supported Authentication](#supported-authentication)).
414
434
@@ -425,7 +445,7 @@ $SPARK_HOME/bin/spark-shell \
425
445
426
446
**Example 2: Workload Identity (AKS)**
427
447
428
-
In an AKS pod with Workload Identity enabled, the `AZURE_*` environment variables injected by the webhook are picked up automatically, so no Comet-specific configuration is required. Setting only the client id and tenant in Hadoop while the webhook supplies `AZURE_FEDERATED_TOKEN_FILE` also works. To set all the values explicitly instead:
448
+
In an AKS pod with Workload Identity enabled, the `AZURE_*` environment variables injected by the webhook are picked up automatically, so no Comet-specific configuration is required. Setting only the client id and tenant in Hadoop while the webhook supplies `AZURE_FEDERATED_TOKEN_FILE` also works; in that case the token file is the only environment variable read. To set all the values explicitly instead:
0 commit comments