Skip to content

Commit c127178

Browse files
committed
fix: read only the Hadoop keys of the auth mechanism the account selects
Hadoop's `AbfsConfiguration.getAuthType` resolves the account-scoped `fs.azure.account.auth.type` over the global one, and `AzureBlobFileSystemStore.initializeClient` then reads only that mechanism's keys. The native scan translated and validated every credential key instead, so an account that selects `SharedKey` while a global OAuth secret remains in the configuration was rejected for missing the OAuth client id and tenant. An explicit auth type now decides which keys are read: only the selected mechanism's keys are translated into the builder and validated, and the other mechanisms' keys, provider class and unsupported-mechanism keys are ignored as Hadoop ignores them, blank or not. Incomplete active credentials are still rejected, and no auth type at all still reads every key.
1 parent b3c4cee commit c127178

2 files changed

Lines changed: 473 additions & 44 deletions

File tree

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -374,7 +374,7 @@ When the Hadoop keys only name the identity, via `fs.azure.account.oauth2.client
374374
| Workload Identity | `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.msi.tenant`, `fs.azure.account.oauth2.token.file` |
375375
| SAS token | `fs.azure.sas.<container>.<account>`, `fs.azure.sas.fixed.token` |
376376

377-
This authority extends only to the mechanisms Comet recognises. An explicit `fs.azure.account.auth.type` is Hadoop choosing a mechanism, so it is validated against the translated keys rather than ignored, and Hadoop ABFS mechanisms that have no native counterpart are neither translated nor allowed to fall through to the environment or to managed identity. In each of these cases no `AZURE_*` variable is consulted, and when the translated keys do not satisfy the mechanism the scan fails with an error naming the key (and, for a class name, the value) instead:
377+
This authority extends only to the mechanisms Comet recognises. An explicit `fs.azure.account.auth.type` is Hadoop choosing a mechanism, so it is validated against the translated keys rather than ignored. It also decides which keys are read, as it does for Hadoop: only the selected mechanism's keys are translated and validated, so an OAuth client secret left in the configuration under an account-scoped `SharedKey`, or an account key under `OAuth`, is left alone, neither rejected nor passed to the store. Hadoop ABFS mechanisms that have no native counterpart are neither translated nor allowed to fall through to the environment or to managed identity. In each of these cases no `AZURE_*` variable is consulted, and when the translated keys do not satisfy the mechanism the scan fails with an error naming the key (and, for a class name, the value) instead:
378378

379379
| Hadoop key | Outcome in the native scan |
380380
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -408,7 +408,7 @@ Internally, each Hadoop key is translated into a specific `AzureConfigKey` on th
408408
| `fs.azure.sas.<container>.<account>` | `SasKey` |
409409
| `fs.azure.sas.fixed.token` | `SasKey` (lower priority) |
410410

411-
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is validated whenever it is set, with or without `fs.azure.account.auth.type`: a class ending in `MsiTokenProvider` stands alone, `WorkloadIdentityTokenProvider` needs the client id and tenant, `ClientCredsTokenProvider` needs the client secret principal, any other class is an error naming the key and the class, and a blank value is an error naming the key. A class matches only by its exact simple name, bare or as the last segment of a qualified name. It, `fs.azure.account.auth.type`, and the unsupported mechanism keys listed under [Supported Authentication](#supported-authentication) are read only to decide whether Hadoop configures an auth mechanism, and whether it is one the native scan can build. The storage account and container always come from the URL.
411+
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is validated whenever it is set and OAuth is the mechanism read, that is with `fs.azure.account.auth.type=OAuth` or with no auth type at all: a class ending in `MsiTokenProvider` stands alone, `WorkloadIdentityTokenProvider` needs the client id and tenant, `ClientCredsTokenProvider` needs the client secret principal, any other class is an error naming the key and the class, and a blank value is an error naming the key. A class matches only by its exact simple name, bare or as the last segment of a qualified name. It, `fs.azure.account.auth.type`, and the unsupported mechanism keys listed under [Supported Authentication](#supported-authentication) are read only to decide whether Hadoop configures an auth mechanism, and whether it is one the native scan can build. The storage account and container always come from the URL.
412412

413413
### Tenant id resolution
414414

0 commit comments

Comments
 (0)