Skip to content

Commit c0aeca8

Browse files
committed
fix: apply Hadoop's default SharedKey when an account key is set without an auth type
`AbfsConfiguration.getAuthType` falls back to `SharedKey` when no `fs.azure.account.auth.type` is set, and `initializeClient` then reads only the account key. The native scan read every mechanism's keys in that case, so a valid account key beside an unused OAuth client secret was rejected for the missing client id and tenant. An account key entry without an auth type now selects `SharedKey`: the OAuth and SAS keys beside it are neither translated nor validated, a blank key is still reported, `fs.azure.account.keyprovider` is still rejected, and the environment contributes nothing. With neither an auth type nor a key, every mechanism is read as before, so OAuth-only and environment-only configurations keep working.
1 parent 6f77779 commit c0aeca8

2 files changed

Lines changed: 190 additions & 25 deletions

File tree

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -374,7 +374,7 @@ When the Hadoop keys only name the identity, via `fs.azure.account.oauth2.client
374374
| Workload Identity | `fs.azure.account.oauth2.client.id`, `fs.azure.account.oauth2.msi.tenant`, `fs.azure.account.oauth2.token.file` |
375375
| SAS token | `fs.azure.sas.<container>.<account>`, `fs.azure.sas.fixed.token` |
376376

377-
This authority extends only to the mechanisms Comet recognises. An explicit `fs.azure.account.auth.type` is Hadoop choosing a mechanism, so it is validated against the translated keys rather than ignored. It also decides which keys are read, as it does for Hadoop: only the selected mechanism's keys are translated and validated, so an OAuth client secret left in the configuration under an account-scoped `SharedKey`, or an account key under `OAuth`, is left alone, neither rejected nor passed to the store. Hadoop ABFS mechanisms that have no native counterpart are neither translated nor allowed to fall through to the environment or to managed identity. In each of these cases no `AZURE_*` variable is consulted, and when the translated keys do not satisfy the mechanism the scan fails with an error naming the key (and, for a class name, the value) instead:
377+
This authority extends only to the mechanisms Comet recognises. An explicit `fs.azure.account.auth.type` is Hadoop choosing a mechanism, so it is validated against the translated keys rather than ignored. It also decides which keys are read, as it does for Hadoop: only the selected mechanism's keys are translated and validated, so an OAuth client secret left in the configuration under an account-scoped `SharedKey`, or an account key under `OAuth`, is left alone, neither rejected nor passed to the store. With no auth type at all, Hadoop defaults to `SharedKey` whenever `fs.azure.account.key` (or its account-scoped form) is present, and the native scan follows suit: only the account key is read, a blank key is still an error, and OAuth or SAS keys beside it are skipped in the same way. With neither an auth type nor an account key, every mechanism's keys are read, as described above. Hadoop ABFS mechanisms that have no native counterpart are neither translated nor allowed to fall through to the environment or to managed identity. In each of these cases no `AZURE_*` variable is consulted, and when the translated keys do not satisfy the mechanism the scan fails with an error naming the key (and, for a class name, the value) instead:
378378

379379
| Hadoop key | Outcome in the native scan |
380380
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -408,7 +408,7 @@ Internally, each Hadoop key is translated into a specific `AzureConfigKey` on th
408408
| `fs.azure.sas.<container>.<account>` | `SasKey` |
409409
| `fs.azure.sas.fixed.token` | `SasKey` (lower priority) |
410410

411-
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is validated whenever it is set and OAuth is the mechanism read, that is with `fs.azure.account.auth.type=OAuth` or with no auth type at all: a class ending in `MsiTokenProvider` stands alone, `WorkloadIdentityTokenProvider` needs the client id and tenant, `ClientCredsTokenProvider` needs the client secret principal, any other class is an error naming the key and the class, and a blank value is an error naming the key. A class matches only by its exact simple name, bare or as the last segment of a qualified name. It, `fs.azure.account.auth.type`, and the unsupported mechanism keys listed under [Supported Authentication](#supported-authentication) are read only to decide whether Hadoop configures an auth mechanism, and whether it is one the native scan can build. The storage account and container always come from the URL.
411+
Hadoop keys beyond these are not translated. `fs.azure.account.oauth.provider.type` is validated whenever it is set and OAuth is the mechanism read, that is with `fs.azure.account.auth.type=OAuth`, or with no auth type and no `fs.azure.account.key` (a key with no auth type selects `SharedKey`, Hadoop's default, and the class is then ignored): a class ending in `MsiTokenProvider` stands alone, `WorkloadIdentityTokenProvider` needs the client id and tenant, `ClientCredsTokenProvider` needs the client secret principal, any other class is an error naming the key and the class, and a blank value is an error naming the key. A class matches only by its exact simple name, bare or as the last segment of a qualified name. It, `fs.azure.account.auth.type`, and the unsupported mechanism keys listed under [Supported Authentication](#supported-authentication) are read only to decide whether Hadoop configures an auth mechanism, and whether it is one the native scan can build. The storage account and container always come from the URL.
412412

413413
### Tenant id resolution
414414

‎native/core/src/parquet/objectstore/azure.rs‎

Lines changed: 188 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -64,18 +64,21 @@
6464
//! credentials provider classes, or no class with a secret or token file), `SAS` needs
6565
//! a SAS token, `Custom` and any other value are errors. The auth type also decides
6666
//! which keys are read, as it does for Hadoop's `initializeClient`: only the selected
67-
//! mechanism's keys are translated and validated, so an inactive global OAuth secret
67+
//! mechanism's keys are translated and validated, so an unused global OAuth secret
6868
//! under an account-scoped `SharedKey`, or an account key under `OAuth`, is ignored
69-
//! rather than rejected or handed to the builder. A provider class is validated
70-
//! whenever it is set, provided OAuth is the mechanism read (with or without an explicit
71-
//! auth type): MSI stands alone, Workload
72-
//! Identity needs the client id and tenant, client credentials need the secret, and
73-
//! any other class is an error. Keys that select a mechanism
74-
//! with no native counterpart (a SAS token provider class, an account key provider
75-
//! class, a refresh token, or a user name or password) are errors too. In every one of
76-
//! these cases the environment is not consulted either, so the scan can neither borrow
77-
//! an ambient identity nor fall back to managed identity in place of the one Hadoop
78-
//! was told to use.
69+
//! rather than rejected or handed to the builder. With no auth type at all, Hadoop's
70+
//! `getAuthType` defaults to `SharedKey`, so an account key entry (blank or not)
71+
//! selects that mechanism the same way and only the key is read; a blank key is still
72+
//! an error. With neither an auth type nor a key, every mechanism's keys are read, as
73+
//! described in points 1 and 2. A provider class is validated whenever it is set,
74+
//! provided OAuth is the mechanism read (with or without an explicit auth type): MSI
75+
//! stands alone, Workload Identity needs the client id and tenant, client credentials
76+
//! need the secret, and any other class is an error. Keys that select a mechanism with
77+
//! no native counterpart (a SAS token provider class, an account key provider class, a
78+
//! refresh token, or a user name or password) are errors too. In every one of these
79+
//! cases the environment is not consulted either, so the scan can neither borrow an
80+
//! ambient identity nor fall back to managed identity in place of the one Hadoop was
81+
//! told to use.
7982
//!
8083
//! Within the Hadoop keys, the account-scoped variant
8184
//! (`fs.azure.account.X.<account>.dfs.core.windows.net`) wins over the global one
@@ -193,7 +196,7 @@ enum AuthMechanism {
193196
/// The mechanism an explicit `fs.azure.account.auth.type` selects for the account, the way
194197
/// Hadoop's `AbfsConfiguration.getAuthType` resolves it: the account-scoped key over the
195198
/// global one. `None` when no auth type is set, or when it names something the scan cannot
196-
/// build (`auth_type_problem` reports that), in which case every Hadoop key is read.
199+
/// build (`auth_type_problem` reports that).
197200
fn explicit_mechanism(
198201
configs: &HashMap<String, String>,
199202
account: Option<&str>,
@@ -211,17 +214,31 @@ fn explicit_mechanism(
211214
}
212215
}
213216

214-
/// Whether Hadoop reads the keys of `mechanism` for this account: every mechanism when no
215-
/// auth type is set, otherwise only the selected one. Hadoop's
217+
/// The mechanism Hadoop selects for the account: the explicit auth type when one is set,
218+
/// otherwise `SharedKey` when an account key entry is present, since that is the default
219+
/// `AbfsConfiguration.getAuthType` falls back to. `None` when no auth type is set and no
220+
/// key is present, in which case every Hadoop key is read. A blank key still counts as
221+
/// present, so it is reported as blank rather than skipped in favour of another mechanism.
222+
fn selected_mechanism(
223+
configs: &HashMap<String, String>,
224+
account: Option<&str>,
225+
) -> Option<AuthMechanism> {
226+
explicit_mechanism(configs, account).or_else(|| {
227+
account_scoped_entry(configs, HADOOP_KEY, account).map(|_| AuthMechanism::SharedKey)
228+
})
229+
}
230+
231+
/// Whether Hadoop reads the keys of `mechanism` for this account: every mechanism when
232+
/// none is selected, otherwise only the selected one. Hadoop's
216233
/// `AzureBlobFileSystemStore.initializeClient` reads only the selected mechanism's keys,
217-
/// so an inactive one (a global OAuth secret under an account-scoped `SharedKey`) is
218-
/// neither translated nor validated.
234+
/// so an unused one (a global OAuth secret under an account-scoped `SharedKey`, or beside
235+
/// an account key with no auth type) is neither translated nor validated.
219236
fn mechanism_is_read(
220237
configs: &HashMap<String, String>,
221238
account: Option<&str>,
222239
mechanism: AuthMechanism,
223240
) -> bool {
224-
explicit_mechanism(configs, account).is_none_or(|selected| selected == mechanism)
241+
selected_mechanism(configs, account).is_none_or(|selected| selected == mechanism)
225242
}
226243

227244
/// The OAuth provider class entry, but only while Hadoop is reading OAuth keys.
@@ -2329,25 +2346,173 @@ mod tests {
23292346
}
23302347

23312348
#[test]
2332-
fn unset_auth_type_still_validates_every_mechanism() {
2333-
// Like `create_store_rejects_client_secret_without_client_id_and_tenant`, with a
2334-
// valid account key alongside, to show its presence does not let an incomplete
2335-
// client secret through when no `auth.type` selects a mechanism.
2349+
fn default_shared_key_ignores_unused_global_oauth_secret() {
2350+
// No auth type, an account-scoped key and a global client secret left over from
2351+
// another account. Hadoop defaults to SharedKey and reads only the key, so the
2352+
// secret is neither handed to the builder nor checked for a client id and tenant.
23362353
let configs = hadoop(&[
23372354
(
2338-
"fs.azure.account.key",
2355+
"fs.azure.account.key.myacct.dfs.core.windows.net",
23392356
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
23402357
),
23412358
("fs.azure.account.oauth2.client.secret", "hadoop-secret"),
2359+
(
2360+
"fs.azure.account.oauth.provider.type",
2361+
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
2362+
),
2363+
]);
2364+
let u = url("abfss://[email protected]/path/file.parquet");
2365+
create_store_with_env(&u, &configs, env_of(&[])).expect("store builds");
2366+
2367+
let builder = builder_for(&configs, &[]);
2368+
assert_eq!(
2369+
value(&builder, AzureConfigKey::AccessKey).as_deref(),
2370+
Some("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=")
2371+
);
2372+
assert_eq!(value(&builder, AzureConfigKey::ClientSecret), None);
2373+
assert_eq!(value(&builder, AzureConfigKey::ClientId), None);
2374+
assert_eq!(value(&builder, AzureConfigKey::AuthorityId), None);
2375+
}
2376+
2377+
#[test]
2378+
fn default_shared_key_ignores_unused_sas_token_and_sas_provider_class() {
2379+
let configs = hadoop(&[
2380+
(
2381+
"fs.azure.account.key",
2382+
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
2383+
),
2384+
("fs.azure.sas.fixed.token", "sv=2020&sig=abc"),
2385+
(
2386+
"fs.azure.sas.token.provider.type",
2387+
"com.example.SasProvider",
2388+
),
23422389
]);
2390+
let u = url("abfss://[email protected]/path/file.parquet");
2391+
create_store_with_env(&u, &configs, env_of(&[])).expect("store builds");
2392+
2393+
let builder = builder_for(&configs, &[]);
2394+
assert_eq!(value(&builder, AzureConfigKey::SasKey), None);
2395+
assert_eq!(
2396+
value(&builder, AzureConfigKey::AccessKey).as_deref(),
2397+
Some("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=")
2398+
);
2399+
}
2400+
2401+
#[test]
2402+
fn default_shared_key_reads_nothing_from_the_environment() {
2403+
let configs = hadoop(&[("fs.azure.account.key", "secret==")]);
2404+
let builder = builder_for(
2405+
&configs,
2406+
&[
2407+
("AZURE_STORAGE_TOKEN", "ambient"),
2408+
("AZURE_CLIENT_ID", "ambient-client"),
2409+
(
2410+
"AZURE_FEDERATED_TOKEN_FILE",
2411+
"/var/run/secrets/azure/tokens/token",
2412+
),
2413+
(
2414+
"IDENTITY_ENDPOINT",
2415+
"http://169.254.169.254/metadata/identity",
2416+
),
2417+
],
2418+
);
2419+
assert_eq!(
2420+
value(&builder, AzureConfigKey::AccessKey).as_deref(),
2421+
Some("secret==")
2422+
);
2423+
assert_eq!(value(&builder, AzureConfigKey::Token), None);
2424+
assert_eq!(value(&builder, AzureConfigKey::ClientId), None);
2425+
assert_eq!(value(&builder, AzureConfigKey::FederatedTokenFile), None);
2426+
assert_eq!(value(&builder, AzureConfigKey::MsiEndpoint), None);
2427+
}
2428+
2429+
#[test]
2430+
fn default_shared_key_still_rejects_blank_account_key() {
2431+
// A key entry selects SharedKey whatever its value, so a blank key is an error on
2432+
// that mechanism rather than a fall-through to the complete OAuth principal.
2433+
let mut pairs = vec![("fs.azure.account.key", "")];
2434+
pairs.extend_from_slice(CLIENT_SECRET_PRINCIPAL);
2435+
let configs = hadoop(&pairs);
23432436
let err = err_of(&configs);
23442437
assert!(
2345-
err.contains("fs.azure.account.oauth2.client.id"),
2438+
err.contains("`fs.azure.account.key` is blank"),
23462439
"unexpected error: {err}"
23472440
);
23482441
assert_hides(&err, &["hadoop-secret"]);
23492442
}
23502443

2444+
#[test]
2445+
fn default_shared_key_ignores_blank_unused_secret() {
2446+
// With a key and no auth type, an OAuth key is never read, so a blank one is no
2447+
// error, just as under an explicit `SharedKey`.
2448+
let configs = hadoop(&[
2449+
(
2450+
"fs.azure.account.key",
2451+
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
2452+
),
2453+
("fs.azure.account.oauth2.client.secret", ""),
2454+
]);
2455+
let u = url("abfss://[email protected]/path/file.parquet");
2456+
create_store_with_env(&u, &configs, env_of(&[])).expect("store builds");
2457+
2458+
let builder = builder_for(&configs, &[]);
2459+
assert_eq!(
2460+
value(&builder, AzureConfigKey::AccessKey).as_deref(),
2461+
Some("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=")
2462+
);
2463+
assert_eq!(value(&builder, AzureConfigKey::ClientSecret), None);
2464+
}
2465+
2466+
#[test]
2467+
fn default_shared_key_ignores_blank_unused_sas_token() {
2468+
let configs = hadoop(&[
2469+
(
2470+
"fs.azure.account.key",
2471+
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
2472+
),
2473+
("fs.azure.sas.fixed.token", ""),
2474+
]);
2475+
let u = url("abfss://[email protected]/path/file.parquet");
2476+
create_store_with_env(&u, &configs, env_of(&[])).expect("store builds");
2477+
2478+
let builder = builder_for(&configs, &[]);
2479+
assert_eq!(value(&builder, AzureConfigKey::SasKey), None);
2480+
assert_eq!(
2481+
value(&builder, AzureConfigKey::AccessKey).as_deref(),
2482+
Some("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=")
2483+
);
2484+
}
2485+
2486+
#[test]
2487+
fn default_shared_key_still_rejects_key_provider_class() {
2488+
let configs = hadoop(&[
2489+
(
2490+
"fs.azure.account.key",
2491+
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
2492+
),
2493+
("fs.azure.account.keyprovider", "com.example.KeyProvider"),
2494+
]);
2495+
let err = err_of(&configs);
2496+
assert!(
2497+
err.contains("fs.azure.account.keyprovider"),
2498+
"unexpected error: {err}"
2499+
);
2500+
}
2501+
2502+
#[test]
2503+
fn no_key_and_no_auth_type_still_reads_every_mechanism() {
2504+
let configs = hadoop(CLIENT_SECRET_PRINCIPAL);
2505+
let u = url("abfss://[email protected]/path/file.parquet");
2506+
create_store_with_env(&u, &configs, env_of(&[])).expect("store builds");
2507+
2508+
let builder = builder_for(&configs, &[]);
2509+
assert_eq!(
2510+
value(&builder, AzureConfigKey::ClientSecret).as_deref(),
2511+
Some("hadoop-secret")
2512+
);
2513+
assert_eq!(value(&builder, AzureConfigKey::AccessKey), None);
2514+
}
2515+
23512516
#[test]
23522517
fn explicit_oauth_scoped_over_global_shared_key_ignores_account_key() {
23532518
let mut pairs = vec![

0 commit comments

Comments
 (0)