Skip to content

Commit 2b044b4

Browse files
committed
fix: validate every Hadoop Azure auth mechanism and borrow only the token file for a named identity
Reject Hadoop configuration the native scan cannot honour instead of letting object_store fall through to the Azure CLI or managed identity: a partial OAuth principal, an explicit auth.type the translated keys do not satisfy, a provider class other than the MSI, workload identity or client credentials providers (matched by exact simple name), the untranslated SAS provider, key provider, refresh token and user name or password keys, and blank values. Translate fs.azure.sas.fixed.token below the container-scoped SAS key, take the account from the host lowercased, and when Hadoop names only an identity borrow AZURE_FEDERATED_TOKEN_FILE alone, erroring when no token file exists. Every error carries one prefix and names the key, never a credential value.
1 parent a7bdbe4 commit 2b044b4

2 files changed

Lines changed: 1342 additions & 108 deletions

File tree

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 39 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -352,43 +352,63 @@ Comet's Parquet scan reads Azure Data Lake Storage Gen2 (ADLS Gen2) through the
352352
[`object_store` crate](https://crates.io/crates/object_store), using the `abfs` and `abfss` URL schemes.
353353
As with S3, standard Hadoop ABFS configurations (the `fs.azure.*` keys you already set in `core-site.xml` or via `spark.hadoop.*`) are translated into the `object_store` crate's format, so existing configurations continue to work.
354354

355-
URLs use the same shape Spark and Hadoop emit: `abfss://<container>@<account>.dfs.core.windows.net/<path>`. The account is taken from the host and the container from the URL user-info. The `wasb`, `wasbs`, `az`, `azure`, and `adl` schemes are not supported by the native scan.
355+
URLs use the same shape Spark and Hadoop emit: `abfss://<container>@<account>.dfs.core.windows.net/<path>`. The account is taken from the host and lowercased before any key lookup, so `MyAcct.dfs.core.windows.net` resolves `fs.azure.account.key.myacct.dfs.core.windows.net`; the container is taken from the URL user-info as written, which matches Hadoop, and Azure requires container names to be lowercase anyway. The `wasb`, `wasbs`, `az`, `azure`, and `adl` schemes are not supported by the native scan.
356356

357357
### Root CA Certificates
358358

359359
Azure scans discover Root CA Certificates the same way S3 scans do. See [Root CA Certificates](#root-ca-certificates) above. The Rust-based scan uses system Root CA Certificates rather than the Java Trust Store.
360360

361361
### Supported Authentication
362362

363-
The Hadoop `fs.azure.*` configuration is authoritative for authentication, matching Hadoop's own ABFS driver, which reads no environment variables. When the Hadoop keys configure a complete auth mechanism (a shared account key, a SAS token, a client secret, a federated token file, or an MSI endpoint), or `fs.azure.account.oauth.provider.type` names `MsiTokenProvider`, that mechanism alone determines the identity and no `AZURE_*` environment variable is consulted: not credentials such as `AZURE_STORAGE_TOKEN`, `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_FEDERATED_TOKEN_FILE`, and not transport settings such as `AZURE_ALLOW_HTTP`, `AZURE_PROXY_URL`, or `AZURE_STORAGE_ENDPOINT`, so the configured identity cannot be outranked, redirected, or intercepted by the environment. A partial Hadoop mechanism, such as `fs.azure.account.oauth2.token.file` without a client id and tenant, is not completed from the environment either; Hadoop requires the client id and tenant as well, and `object_store` then falls through its credential chain to the node's managed identity, so configure the full set.
363+
The Hadoop `fs.azure.*` configuration is authoritative for authentication, matching Hadoop's own ABFS driver, which reads no environment variables. When the Hadoop keys configure a complete auth mechanism (a shared account key, a SAS token, a client secret, a federated token file, or an MSI endpoint), or `fs.azure.account.oauth.provider.type` names `MsiTokenProvider`, that mechanism alone determines the identity and no `AZURE_*` environment variable is consulted: not credentials such as `AZURE_STORAGE_TOKEN`, `AZURE_STORAGE_ACCOUNT_KEY`, or `AZURE_FEDERATED_TOKEN_FILE`, and not transport settings such as `AZURE_ALLOW_HTTP`, `AZURE_PROXY_URL`, or `AZURE_STORAGE_ENDPOINT`, so the configured identity cannot be outranked, redirected, or intercepted by the environment. A partial Hadoop mechanism, such as `fs.azure.account.oauth2.token.file` or `fs.azure.account.oauth2.client.secret` without a client id and tenant, is not completed from the environment either: the scan fails with an error naming the missing keys, because Hadoop requires the client id and tenant as well and `object_store` would otherwise fall through its credential chain to the node's managed identity. A blank SAS token or credential value (`fs.azure.account.key`, `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.client.secret`, `fs.azure.account.oauth2.msi.tenant`, `fs.azure.account.oauth2.token.file`, and the other translated keys) is an error for the same reason, naming the exact key: `object_store` would send unsigned requests or fall through its chain.
364364

365-
When the Hadoop keys configure no auth mechanism, or only name the identity via `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.msi.tenant`, or `fs.azure.account.oauth2.msi.authority`, the `AZURE_*` environment variables (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_AUTHORITY_HOST`, `AZURE_STORAGE_*`) are honored and the Hadoop keys are applied on top. This is what makes AKS Workload Identity work in a stock pod with no extra configuration. Transport settings from the environment apply only in this case, alongside the environment credentials.
365+
When the Hadoop keys configure no auth mechanism and name no identity, the `AZURE_*` environment variables (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_AUTHORITY_HOST`, `AZURE_STORAGE_*`) are honored. This is what makes AKS Workload Identity work in a stock pod with no extra configuration. Transport settings from the environment apply only in this case, alongside the environment credentials. The same rule covers `AZURE_USE_AZURE_CLI`, which makes `object_store` take its token from a logged-in Azure CLI, and `AZURE_SKIP_SIGNATURE`, which makes it send unsigned requests for public containers: both are honored only when Hadoop supplies neither a mechanism nor an identity, like every other `AZURE_*` variable.
366+
367+
When the Hadoop keys only name the identity, via `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.msi.tenant`, or `fs.azure.account.oauth2.msi.authority`, or via `fs.azure.account.oauth.provider.type` ending in `WorkloadIdentityTokenProvider` together with the client id and tenant, only `AZURE_FEDERATED_TOKEN_FILE` is read from the environment, and only when `fs.azure.account.oauth2.token.file` is not set. No other `AZURE_*` variable is consulted, so an ambient account key or bearer token cannot outrank the named principal. A named principal with no token file in either place is an error naming both sources, and a borrowed token file still needs the client id and tenant from the Hadoop keys.
366368

367369
| Authentication method | Hadoop keys |
368370
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
369371
| Shared account key | `fs.azure.account.key` |
370372
| OAuth2 client credentials | `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.client.secret`, `fs.azure.account.oauth2.client.endpoint` (tenant id is extracted from the endpoint URL) |
371373
| Managed identity (MSI) | `fs.azure.account.oauth2.msi.tenant`, `fs.azure.account.oauth2.msi.endpoint`, `fs.azure.account.oauth2.msi.authority` |
372374
| Workload Identity | `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.msi.tenant`, `fs.azure.account.oauth2.token.file` |
373-
| SAS token | `fs.azure.sas.<container>.<account>` |
375+
| SAS token | `fs.azure.sas.<container>.<account>`, `fs.azure.sas.fixed.token` |
376+
377+
This authority extends only to the mechanisms Comet recognises. An explicit `fs.azure.account.auth.type` is Hadoop choosing a mechanism, so it is validated against the translated keys rather than ignored, and Hadoop ABFS mechanisms that have no native counterpart are neither translated nor allowed to fall through to the environment or to managed identity. In each of these cases no `AZURE_*` variable is consulted, and when the translated keys do not satisfy the mechanism the scan fails with an error naming the key (and, for a class name, the value) instead:
378+
379+
| Hadoop key | Outcome in the native scan |
380+
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
381+
| `fs.azure.account.auth.type` = `SharedKey` | Accepted only with `fs.azure.account.key` |
382+
| `fs.azure.account.auth.type` = `OAuth` | Accepted with `fs.azure.account.oauth.provider.type` ending in `MsiTokenProvider`, `WorkloadIdentityTokenProvider` (with the client id and tenant, and a token file from `fs.azure.account.oauth2.token.file` or `AZURE_FEDERATED_TOKEN_FILE`, the only environment variable read in that case), or `ClientCredsTokenProvider` (with the client secret), or with no provider class and a client secret or token file; any other provider class is rejected, naming the class |
383+
| `fs.azure.account.auth.type` = `Custom` | Loads a custom token provider class, which the native scan cannot run |
384+
| `fs.azure.account.auth.type` = `SAS` | Rejected unless `fs.azure.sas.fixed.token` (or a container-scoped SAS token) is also set |
385+
| `fs.azure.account.auth.type` = anything else | Rejected, naming the value; the native scan supports `SharedKey`, `OAuth` and `SAS` |
386+
| `fs.azure.sas.token.provider.type` | Loads a SAS token provider class |
387+
| `fs.azure.account.keyprovider` | Loads an account key provider class |
388+
| `fs.azure.account.oauth2.refresh.token` | Refresh-token OAuth flow |
389+
| `fs.azure.account.oauth2.user.name` | User name and password OAuth flow |
390+
| `fs.azure.account.oauth2.user.password` | User name and password OAuth flow |
391+
392+
The account-scoped forms of these keys (`<key>.<account>.dfs.core.windows.net` and so on) are rejected the same way.
374393

375394
### Hadoop-to-object_store key mapping
376395

377396
Internally, each Hadoop key is translated into a specific `AzureConfigKey` on the underlying `object_store` `MicrosoftAzureBuilder`:
378397

379-
| Hadoop key (account-scoped suffix omitted) | `AzureConfigKey` |
380-
| ------------------------------------------ | ------------------------ |
381-
| `fs.azure.account.key` | `AccessKey` |
382-
| `fs.azure.account.oauth2.client.id` | `ClientId` |
383-
| `fs.azure.account.oauth2.client.secret` | `ClientSecret` |
384-
| `fs.azure.account.oauth2.client.endpoint` | `AuthorityId` (from URL) |
385-
| `fs.azure.account.oauth2.msi.tenant` | `AuthorityId` |
386-
| `fs.azure.account.oauth2.msi.endpoint` | `MsiEndpoint` |
387-
| `fs.azure.account.oauth2.msi.authority` | `AuthorityHost` |
388-
| `fs.azure.account.oauth2.token.file` | `FederatedTokenFile` |
389-
| `fs.azure.sas.<container>.<account>` | `SasKey` |
390-
391-
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is read only to decide whether Hadoop configures an auth mechanism. The storage account and container always come from the URL.
398+
| Hadoop key (account-scoped suffix omitted) | `AzureConfigKey` |
399+
| ------------------------------------------ | ------------------------- |
400+
| `fs.azure.account.key` | `AccessKey` |
401+
| `fs.azure.account.oauth2.client.id` | `ClientId` |
402+
| `fs.azure.account.oauth2.client.secret` | `ClientSecret` |
403+
| `fs.azure.account.oauth2.client.endpoint` | `AuthorityId` (from URL) |
404+
| `fs.azure.account.oauth2.msi.tenant` | `AuthorityId` |
405+
| `fs.azure.account.oauth2.msi.endpoint` | `MsiEndpoint` |
406+
| `fs.azure.account.oauth2.msi.authority` | `AuthorityHost` |
407+
| `fs.azure.account.oauth2.token.file` | `FederatedTokenFile` |
408+
| `fs.azure.sas.<container>.<account>` | `SasKey` |
409+
| `fs.azure.sas.fixed.token` | `SasKey` (lower priority) |
410+
411+
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is validated whenever it is set, with or without `fs.azure.account.auth.type`: a class ending in `MsiTokenProvider` stands alone, `WorkloadIdentityTokenProvider` needs the client id and tenant, `ClientCredsTokenProvider` needs the client secret principal, any other class is an error naming the key and the class, and a blank value is an error naming the key. A class matches only by its exact simple name, bare or as the last segment of a qualified name. It, `fs.azure.account.auth.type`, and the unsupported mechanism keys listed under [Supported Authentication](#supported-authentication) are read only to decide whether Hadoop configures an auth mechanism, and whether it is one the native scan can build. The storage account and container always come from the URL.
392412

393413
### Tenant id resolution
394414

@@ -408,7 +428,7 @@ Account-scoped keys take precedence over global ones, mirroring Hadoop ABFS's ow
408428
3. `<key>.<account>`
409429
4. `<key>` (unscoped / global)
410430

411-
The SAS namespace follows the same shape with the container inlined into the key name: `fs.azure.sas.<container>.<account>.dfs.core.windows.net`, then `fs.azure.sas.<container>.<account>.blob.core.windows.net`, then `fs.azure.sas.<container>.<account>`.
431+
The SAS namespace follows the same shape with the container inlined into the key name: `fs.azure.sas.<container>.<account>.dfs.core.windows.net`, then `fs.azure.sas.<container>.<account>.blob.core.windows.net`, then `fs.azure.sas.<container>.<account>`. When none of those is set, the account-level `fs.azure.sas.fixed.token` is probed in the account-scoped order above, so a container-scoped token always wins over a fixed token. A SAS token that is empty or only whitespace is rejected rather than treated as unset, and a blank container-scoped key is rejected even when a valid `fs.azure.sas.fixed.token` is set: the fixed token is never used as a fallback for it.
412432

413433
A Hadoop value always wins over an `AZURE_*` environment variable for the same key. When the Hadoop keys configure an auth mechanism, the environment's credential variables are not consulted at all (see [Supported Authentication](#supported-authentication)).
414434

@@ -425,7 +445,7 @@ $SPARK_HOME/bin/spark-shell \
425445

426446
**Example 2: Workload Identity (AKS)**
427447

428-
In an AKS pod with Workload Identity enabled, the `AZURE_*` environment variables injected by the webhook are picked up automatically, so no Comet-specific configuration is required. Setting only the client id and tenant in Hadoop while the webhook supplies `AZURE_FEDERATED_TOKEN_FILE` also works. To set all the values explicitly instead:
448+
In an AKS pod with Workload Identity enabled, the `AZURE_*` environment variables injected by the webhook are picked up automatically, so no Comet-specific configuration is required. Setting only the client id and tenant in Hadoop while the webhook supplies `AZURE_FEDERATED_TOKEN_FILE` also works; in that case the token file is the only environment variable read. To set all the values explicitly instead:
429449

430450
```shell
431451
$SPARK_HOME/bin/spark-shell \

0 commit comments

Comments
 (0)