Skip to content

Commit 61fbbe2

Browse files
authored
fix: fall back to Spark when the driver cannot use the ABFS authentication (#7221)
* fix: fall back to Spark when the driver cannot use the ABFS authentication A native ABFS scan whose authentication Hadoop declines, resolves to a RefreshToken or UserPassword OAuth provider, or fails to resolve on the driver now falls back to Spark's own scan instead of planning natively and failing every task. Closes #7202 * fix: keep a URL password out of the ABFS fallback reason * fix: keep a URL password out of Hadoop messages in the ABFS fallback reason Strip the password from the user info wherever a Failed message quotes it, cover the supported OAuth and SAS mechanisms staying native, and add end-to-end fallback tests for Parquet and CSV scans. * fix: fall back for a client credentials endpoint the native ABFS store rejects nativeFallbackReason now declines a Resolved ClientCredsTokenProvider whose endpoint is not https or has no tenant segment, which azure.rs rejects, so the scan reads through Spark instead of failing every task. The reason leaves the endpoint out because it can carry user info. Also reuse resolveS3CompliantSchemes in objectStoreOptionsAndAliasOptions. * fix: fall back for an http Workload Identity authority in the native ABFS scan object_store's token client refuses http, so a Resolved WorkloadIdentityTokenProvider whose fs.azure.account.oauth2.msi.authority is not https plans natively and fails every task, while Hadoop posts to it. nativeFallbackReason now declines it. The reason leaves the authority out because it can carry user info. * fix: ask native whether it can build the ABFS store instead of copying its rules The driver now calls NativeBase.abfsStoreError, which runs azure::create_store on the resolved options, and falls back with native's own message when it fails. The Scala copies of the client credentials endpoint and Workload Identity authority rules are removed. Native rejects an http endpoint or authority when it builds the store, so native writes get that error up front too. * fix: fall back when asking native about the ABFS store fails, and type the probe's arrays Only a LinkageError (the library is not loaded) assumes the store builds. Any other failure from the probe, such as a panic while building the store, would fail every task of a native scan, so it is now a fallback reason. The JNI function takes JObjectArray<JString>, which drops the unsafe JString casts.
1 parent 967d09b commit 61fbbe2

8 files changed

Lines changed: 512 additions & 26 deletions

File tree

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -462,7 +462,7 @@ For each `abfs` or `abfss` path, the Spark driver asks the hadoop-azure library
462462

463463
The native scan receives the resolved values and builds the `object_store` client from them. It reads no `fs.azure.*` key of its own. A scan whose files span more than one ABFS container or account, or that mixes `abfs`/`abfss` paths with other schemes, falls back to Spark's own scan.
464464

465-
hadoop-azure must be on the driver classpath. It already is whenever Spark can list the path. When it is missing, the scan fails with an error that says so.
465+
hadoop-azure must be on the driver classpath. It already is whenever Spark can list the path. When the driver cannot load it, or its `AbfsConfiguration` does not match the versions above (a vendor fork, for example), the scan falls back to Spark's own scan.
466466

467467
#### Mechanisms
468468

@@ -479,15 +479,15 @@ hadoop-azure must be on the driver classpath. It already is whenever Spark can l
479479
| `WorkloadIdentityTokenProvider` with a `fs.azure.account.oauth2.client.assertion.provider.type` (hadoop-azure 3.5.0 and later) | Not supported | |
480480
| `UserboundSASWithOAuth` (hadoop-azure 3.5.0 and later) | Not supported | |
481481

482-
Custom provider classes hand each token request to Java code the native scan cannot call, and the refresh token and user password flows have no `object_store` counterpart. When Hadoop selects one of these, the scan fails with an error naming the auth type and, where one is configured, the class. Configure a supported mechanism for that account, or keep the path on Spark's own scan. Values arrive as Hadoop resolved them, trimmed and defaulted where the ABFS driver trims and defaults them, so the store sees what the driver would have used.
482+
Custom provider classes hand each token request to Java code the native scan cannot call, and the refresh token and user password flows have no `object_store` counterpart. When Hadoop selects one of these, the scan falls back to Spark's own scan, which authenticates through hadoop-azure, and the fallback reason names the auth type and, for SAS, OAuth and Workload Identity, the provider class (`Custom` names only the auth type). Configure a supported mechanism for that account to read it natively. Values arrive as Hadoop resolved them, trimmed and defaulted where the ABFS driver trims and defaults them, so the store sees what the driver would have used.
483483

484-
Client credentials and Workload Identity token requests go over HTTPS only. The managed identity request goes to the IMDS endpoint over HTTP, as Hadoop's does. For client credentials, Hadoop posts to `fs.azure.account.oauth2.client.endpoint` as given, while `object_store` posts to `<authority host>/<tenant>/oauth2/v2.0/token`. Comet takes the tenant from the path segment before the last `oauth2` and everything before that segment as the authority host, so a v1 `/oauth2/token` endpoint and a proxy with a path prefix both resolve. The token request path is always the v2.0 one. An `http://` endpoint fails.
484+
Client credentials and Workload Identity token requests go over HTTPS only. The managed identity request goes to the IMDS endpoint over HTTP, as Hadoop's does. For client credentials, Hadoop posts to `fs.azure.account.oauth2.client.endpoint` as given, while `object_store` posts to `<authority host>/<tenant>/oauth2/v2.0/token`. Comet takes the tenant from the path segment before the last `oauth2` and everything before that segment as the authority host, so a v1 `/oauth2/token` endpoint and a proxy with a path prefix both resolve. The token request path is always the v2.0 one. An `http://` endpoint, or one with no tenant segment, makes the scan fall back to Spark's own scan. So does an `http://` Workload Identity authority (`fs.azure.account.oauth2.msi.authority`). The driver decides this by asking the native store whether it can be built from the resolved values, so the scan falls back exactly when the native store would reject them. A native write to such an account fails with an error naming the key.
485485

486486
### Environment variables
487487

488488
The `AZURE_*` variables (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_FEDERATED_TOKEN_FILE`, `AZURE_STORAGE_ACCOUNT_KEY`, `AZURE_USE_AZURE_CLI`, and the others `object_store` reads) and `IDENTITY_ENDPOINT` are read only when Hadoop configures no authentication for the account at all: none of `fs.azure.account.auth.type`, `fs.azure.account.key`, `fs.azure.account.keyprovider`, `fs.azure.shellkeyprovider.script`, `fs.azure.account.oauth.provider.type`, `fs.azure.sas.token.provider.type`, `fs.azure.sas.fixed.token` or any `fs.azure.account.oauth2.*` key resolves for it. That case is what lets AKS Workload Identity work with no Hadoop configuration for tables whose files Spark does not list through Hadoop, such as Iceberg tables. A plain Parquet path still needs Hadoop configured for Spark's own listing.
489489

490-
Once Hadoop names anything, only the resolved values reach the store. A resolution error (a missing mandatory key, an invalid account key, an unknown provider class) fails the scan with Hadoop's exception. Messages that could quote a credential, such as the key provider's, are kept in the Spark driver log and the scan error names the exception class. The environment is never a fallback. One read happens inside `object_store` rather than Comet: under a resolved managed identity, its IMDS provider picks up `IDENTITY_HEADER` when it fetches a token.
490+
Once Hadoop names anything, only the resolved values reach the store. A resolution error (a missing mandatory key, an invalid account key, an unknown provider class) makes the scan fall back to Spark's own scan, which reports the error in hadoop-azure's own words if it cannot authenticate either. The fallback reason carries Hadoop's exception. Messages that could quote a credential, such as the key provider's, are kept in the Spark driver log and the reason names the exception class. The environment is never a fallback. One read happens inside `object_store` rather than Comet: under a resolved managed identity, its IMDS provider picks up `IDENTITY_HEADER` when it fetches a token.
491491

492492
### Credentials in transit
493493

@@ -557,7 +557,7 @@ Comet 1.1 started from the environment and overlaid the Hadoop keys it found by
557557
- Transport variables such as `AZURE_STORAGE_ENDPOINT` and `AZURE_ALLOW_HTTP` no longer apply beside a Hadoop mechanism. They are read only when Hadoop configures none.
558558
- The WASB key `fs.azure.sas.<container>.<account>`, which the ABFS driver never read, is no longer read by the native scan either. Use `fs.azure.sas.fixed.token`, which the native scan now reads.
559559
- A lower-case `oauth` auth type, or OAuth keys without `fs.azure.account.auth.type`, fail the way they fail for Spark. In the second case Hadoop defaults to `SharedKey` and the key provider fails. On hadoop-azure 3.4.1 and later the driver log names the missing key. 3.3.4 reports only `Failure to initialize configuration`.
560-
- A configured mechanism the native scan cannot build fails with an error instead of silently reading the environment.
560+
- A configured mechanism the native scan cannot build falls back to Spark's own scan instead of silently reading the environment. So does a driver whose hadoop-azure Comet cannot load.
561561

562562
### Azure Limitations
563563

‎native/core/src/lib.rs‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ extern crate core;
3030
extern crate datafusion_comet_jni_bridge;
3131

3232
use jni::{
33-
objects::{JClass, JString},
33+
objects::{JClass, JObjectArray, JString},
3434
sys::{jboolean, jstring, JNI_FALSE},
3535
EnvUnowned,
3636
};
@@ -41,6 +41,7 @@ use log4rs::{
4141
encode::pattern::PatternEncoder,
4242
Config,
4343
};
44+
use std::collections::HashMap;
4445

4546
// Re-export from jvm-bridge crate for internal use
4647
pub use datafusion_comet_jni_bridge::errors;
@@ -319,6 +320,33 @@ pub extern "system" fn Java_org_apache_comet_NativeBase_icebergStorageSchemes(
319320
})
320321
}
321322

323+
/// JNI: why the native scan cannot build its ABFS store for `url` from the object store options
324+
/// `keys`/`values`, or null when it can. The driver asks this when it plans a scan whose ABFS
325+
/// authentication resolved, so a store native would reject falls back to Spark instead of every
326+
/// task failing. See `azure::store_error`.
327+
#[no_mangle]
328+
pub extern "system" fn Java_org_apache_comet_NativeBase_abfsStoreError(
329+
env: EnvUnowned,
330+
_: JClass,
331+
url: JString,
332+
keys: JObjectArray<JString>,
333+
values: JObjectArray<JString>,
334+
) -> jstring {
335+
try_unwrap_or_throw(&env, |env| {
336+
let url: String = url.try_to_string(env)?;
337+
let mut configs = HashMap::new();
338+
for i in 0..keys.len(env)? {
339+
let key = keys.get_element(env, i)?.try_to_string(env)?;
340+
let value = values.get_element(env, i)?.try_to_string(env)?;
341+
configs.insert(key, value);
342+
}
343+
match parquet::objectstore::azure::store_error(&url, &configs) {
344+
Some(message) => Ok(env.new_string(message)?.into_raw()),
345+
None => Ok(std::ptr::null_mut()),
346+
}
347+
})
348+
}
349+
322350
// Creates a default log4rs config, which logs to console with log level.
323351
fn default_logger_config(log_level: &str) -> CometResult<Config> {
324352
let console_append = ConsoleAppender::builder()

‎native/core/src/parquet/objectstore/azure.rs‎

Lines changed: 84 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,11 @@
3737
//! - `error`: an error carrying the driver's exception class and message.
3838
//! - missing or anything else: an error, so a plan that did not run the resolver cannot pass.
3939
//!
40+
//! Scans never reach any of these errors. The driver handles `declined` and `error` itself and,
41+
//! for `resolved`, asks [`store_error`] (through `NativeBase.abfsStoreError`) whether
42+
//! [`create_store`] can build the store the scan would build. A scan that would fail here is
43+
//! planned on Spark instead. Native writes still fail with these errors.
44+
//!
4045
//! `comet.azure.auth.type` is Hadoop's `AuthType` name, `comet.azure.oauth.provider.class` the
4146
//! OAuth provider class, `comet.azure.sas.provider.class` a custom SAS provider and
4247
//! `comet.azure.oauth.assertion.provider.class` a Workload Identity client assertion provider.
@@ -62,7 +67,9 @@
6267
//! the client id and tenant to its request only when they are non-empty. The MSI tenant and
6368
//! authority are accepted but unused: object_store's IMDS provider reads only the client id and
6469
//! endpoint. `RefreshTokenBasedTokenProvider` and `UserPasswordTokenProvider` have no
65-
//! `object_store` counterpart and fail as unsupported. A value a mechanism needs but that is
70+
//! `object_store` counterpart and fail as unsupported. The client credentials endpoint and the
71+
//! Workload Identity authority must be `https` URLs: the store is built without `allow_http`, so
72+
//! an `http` token request would fail on every attempt. A value a mechanism needs but that is
6673
//! absent (or, for the SAS token and the client credentials endpoint, unusable) is an error
6774
//! naming the key; other `fs.azure.*` keys are ignored. Values are applied as forwarded (Hadoop
6875
//! already trimmed and defaulted them), except that the trailing slash Hadoop appends to
@@ -129,6 +136,21 @@ pub fn create_store(
129136
Ok((Box::new(store), path))
130137
}
131138

139+
/// Why [`create_store`] cannot build a store for `url` from `configs`, or `None` when it can.
140+
/// The driver asks this at planning time so that a scan the store would reject falls back to
141+
/// Spark, from the same code the scan runs instead of a copy of its rules. Building makes no
142+
/// request, and the message names keys, never their values.
143+
pub fn store_error(url: &str, configs: &HashMap<String, String>) -> Option<String> {
144+
let url = match Url::parse(url) {
145+
Ok(url) => url,
146+
Err(e) => return Some(format!("invalid ABFS URL: {e}")),
147+
};
148+
create_store(&url, configs).err().map(|e| match e {
149+
object_store::Error::Generic { source, .. } => source.to_string(),
150+
other => other.to_string(),
151+
})
152+
}
153+
132154
/// The builder the markers in `configs` call for. Only `none` touches the environment.
133155
fn builder_for(
134156
url: &Url,
@@ -232,6 +254,7 @@ fn client_creds_settings(
232254
before `oauth2`"
233255
))
234256
})?;
257+
require_https(HADOOP_OAUTH_CLIENT_ENDPOINT, &host)?;
235258
Ok(vec![
236259
(
237260
AzureConfigKey::ClientId,
@@ -274,6 +297,14 @@ fn msi_settings(configs: &HashMap<String, String>) -> Result<Settings, object_st
274297
fn workload_identity_settings(
275298
configs: &HashMap<String, String>,
276299
) -> Result<Settings, object_store::Error> {
300+
let authority = required(configs, HADOOP_MSI_AUTHORITY)?;
301+
let has_host = Url::parse(&authority).is_ok_and(|url| url.has_host());
302+
if !has_host {
303+
return Err(config_error(format!(
304+
"`{HADOOP_MSI_AUTHORITY}` has no `scheme://host` origin"
305+
)));
306+
}
307+
require_https(HADOOP_MSI_AUTHORITY, &authority)?;
277308
Ok(vec![
278309
(
279310
AzureConfigKey::ClientId,
@@ -287,13 +318,23 @@ fn workload_identity_settings(
287318
AzureConfigKey::FederatedTokenFile,
288319
required(configs, HADOOP_WI_TOKEN_FILE)?,
289320
),
290-
(
291-
AzureConfigKey::AuthorityHost,
292-
authority_host(&required(configs, HADOOP_MSI_AUTHORITY)?),
293-
),
321+
(AzureConfigKey::AuthorityHost, authority_host(&authority)),
294322
])
295323
}
296324

325+
/// The store is built without `allow_http`, so a token request to an `http` URL fails on every
326+
/// attempt. Fail at build time instead, naming the key. `url` must already parse.
327+
fn require_https(key: &str, url: &str) -> Result<(), object_store::Error> {
328+
if Url::parse(url).is_ok_and(|url| url.scheme() == "https") {
329+
Ok(())
330+
} else {
331+
Err(config_error(format!(
332+
"`{key}` is not an `https` URL; the native ABFS store sends token requests over \
333+
HTTPS only"
334+
)))
335+
}
336+
}
337+
297338
fn required(configs: &HashMap<String, String>, key: &str) -> Result<String, object_store::Error> {
298339
configs.get(key).cloned().ok_or_else(|| {
299340
config_error(format!(
@@ -529,6 +570,33 @@ mod tests {
529570
assert!(err.contains("Scheme of URL is not Azure"), "{err}");
530571
}
531572

573+
#[test]
574+
fn store_error_answers_whether_create_store_builds() {
575+
let shared_key = with(shared_key(), HADOOP_KEY, KEY);
576+
assert_eq!(store_error(URL, &shared_key), None);
577+
assert_eq!(store_error(URL, &client_creds()), None);
578+
let http = with(
579+
client_creds(),
580+
HADOOP_OAUTH_CLIENT_ENDPOINT,
581+
"http://login.microsoftonline.com/hadoop-tenant/oauth2/token",
582+
);
583+
let err = store_error(URL, &http).expect("an error");
584+
assert!(
585+
err.starts_with(&format!(
586+
"`{HADOOP_OAUTH_CLIENT_ENDPOINT}` is not an `https` URL"
587+
)),
588+
"{err}"
589+
);
590+
let declined = configs(&[
591+
(MARKER_RESOLUTION, "declined"),
592+
(MARKER_AUTH_TYPE, "Custom"),
593+
]);
594+
let err = store_error(URL, &declined).expect("an error");
595+
assert!(err.contains("auth type Custom"), "{err}");
596+
let err = store_error("not a url", &shared_key).expect("an error");
597+
assert!(err.starts_with("invalid ABFS URL"), "{err}");
598+
}
599+
532600
#[test]
533601
fn client_creds_take_tenant_and_authority_host_from_the_endpoint_alone() {
534602
assert_exactly(
@@ -563,9 +631,11 @@ mod tests {
563631
);
564632
// An endpoint without a tenant segment or without a `scheme://host` origin cannot
565633
// fall through to another credential or to the default authority host.
634+
// Nor can an `http` endpoint, which the store would refuse on every token request.
566635
for endpoint in [
567636
"https://auth-proxy.example/",
568637
"file:///hadoop-tenant/oauth2/token",
638+
"http://login.microsoftonline.com/hadoop-tenant/oauth2/token",
569639
] {
570640
let err = err(&with(
571641
client_creds(),
@@ -694,6 +764,15 @@ mod tests {
694764
"{authority}"
695765
);
696766
}
767+
// The token request goes to the authority over a client that refuses `http`.
768+
for authority in [
769+
"http://login.microsoftonline.com/",
770+
"login.microsoftonline.com",
771+
"file:///",
772+
] {
773+
let err = err(&with(workload_identity(), HADOOP_MSI_AUTHORITY, authority));
774+
assert!(err.contains(HADOOP_MSI_AUTHORITY), "{authority}: {err}");
775+
}
697776
}
698777

699778
#[test]

‎spark/src/main/java/org/apache/comet/NativeBase.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -401,4 +401,17 @@ public static void releaseNative() throws Throwable {
401401
* @return the supported schemes joined by commas, e.g. "file,memory,gs,s3,s3a"
402402
*/
403403
public static native String icebergStorageSchemes(boolean forWrite);
404+
405+
/**
406+
* Why the native scan cannot build its ABFS object store for {@code url} from the given object
407+
* store options, or null when it can. This runs the store construction the scan itself runs, so
408+
* the planner falls back on exactly the configurations native rejects rather than a copy of its
409+
* rules. Building the store makes no request, and the message names option keys, never values.
410+
*
411+
* @param url an {@code abfs} or {@code abfss} URL naming the container and account
412+
* @param keys the option keys, as produced by {@code AbfsAuthResolver.toOptions}
413+
* @param values the option values, in the order of {@code keys}
414+
* @return the reason the store cannot be built, or null
415+
*/
416+
public static native String abfsStoreError(String url, String[] keys, String[] values);
404417
}

0 commit comments

Comments
 (0)