Skip to content

Commit 8c7c35d

Browse files
committed
fix: fall back for a client credentials endpoint the native ABFS store rejects
nativeFallbackReason now declines a Resolved ClientCredsTokenProvider whose endpoint is not https or has no tenant segment, which azure.rs rejects, so the scan reads through Spark instead of failing every task. The reason leaves the endpoint out because it can carry user info. Also reuse resolveS3CompliantSchemes in objectStoreOptionsAndAliasOptions.
1 parent c2a7adb commit 8c7c35d

5 files changed

Lines changed: 95 additions & 7 deletions

File tree

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -481,7 +481,7 @@ hadoop-azure must be on the driver classpath. It already is whenever Spark can l
481481

482482
Custom provider classes hand each token request to Java code the native scan cannot call, and the refresh token and user password flows have no `object_store` counterpart. When Hadoop selects one of these, the scan falls back to Spark's own scan, which authenticates through hadoop-azure, and the fallback reason names the auth type and, for SAS, OAuth and Workload Identity, the provider class (`Custom` names only the auth type). Configure a supported mechanism for that account to read it natively. Values arrive as Hadoop resolved them, trimmed and defaulted where the ABFS driver trims and defaults them, so the store sees what the driver would have used.
483483

484-
Client credentials and Workload Identity token requests go over HTTPS only. The managed identity request goes to the IMDS endpoint over HTTP, as Hadoop's does. For client credentials, Hadoop posts to `fs.azure.account.oauth2.client.endpoint` as given, while `object_store` posts to `<authority host>/<tenant>/oauth2/v2.0/token`. Comet takes the tenant from the path segment before the last `oauth2` and everything before that segment as the authority host, so a v1 `/oauth2/token` endpoint and a proxy with a path prefix both resolve. The token request path is always the v2.0 one. An `http://` endpoint fails.
484+
Client credentials and Workload Identity token requests go over HTTPS only. The managed identity request goes to the IMDS endpoint over HTTP, as Hadoop's does. For client credentials, Hadoop posts to `fs.azure.account.oauth2.client.endpoint` as given, while `object_store` posts to `<authority host>/<tenant>/oauth2/v2.0/token`. Comet takes the tenant from the path segment before the last `oauth2` and everything before that segment as the authority host, so a v1 `/oauth2/token` endpoint and a proxy with a path prefix both resolve. The token request path is always the v2.0 one. An `http://` endpoint, or one with no tenant segment, makes the scan fall back to Spark's own scan.
485485

486486
### Environment variables
487487

‎native/core/src/parquet/objectstore/azure.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,9 @@
3737
//! - `error`: an error carrying the driver's exception class and message.
3838
//! - missing or anything else: an error, so a plan that did not run the resolver cannot pass.
3939
//!
40-
//! Scans never reach the `declined` and `error` arms, or the providers `resolved_settings`
41-
//! rejects: the driver plans those on Spark instead. Native writes still fail with them.
40+
//! Scans never reach the `declined` and `error` arms, the providers `resolved_settings` rejects,
41+
//! or a client credentials endpoint that is not `https` or has no tenant segment: the driver plans
42+
//! those on Spark instead. Native writes still fail with them.
4243
//!
4344
//! `comet.azure.auth.type` is Hadoop's `AuthType` name, `comet.azure.oauth.provider.class` the
4445
//! OAuth provider class, `comet.azure.sas.provider.class` a custom SAS provider and

‎spark/src/main/scala/org/apache/comet/objectstore/AbfsAuthResolver.scala‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -472,9 +472,10 @@ private[comet] object AbfsAuthResolver extends Logging {
472472
/**
473473
* Why the native store cannot read `uri` under `outcome`, or None when it can: a mechanism
474474
* Hadoop named that runs Java code per request, a built-in OAuth provider with no
475-
* `object_store` counterpart, or a resolution that failed. All are known on the driver, so the
476-
* scan falls back to Spark's own reader, which authenticates through hadoop-azure, instead of
477-
* planning natively and failing every task.
475+
* `object_store` counterpart, a client credentials endpoint the native store rejects, or a
476+
* resolution that failed. All are known on the driver, so the scan falls back to Spark's own
477+
* reader, which authenticates through hadoop-azure, instead of planning natively and failing
478+
* every task.
478479
*/
479480
def nativeFallbackReason(uri: URI, outcome: Outcome): Option[String] = {
480481
// The container and host only: a password in the user info stays out of the reason.
@@ -496,6 +497,13 @@ private[comet] object AbfsAuthResolver extends Logging {
496497
Some(c @ (Providers.REFRESH_TOKEN | Providers.USER_PASSWORD)),
497498
_) =>
498499
Some(unsupported(s"auth type ${AuthTypes.OAUTH} with provider $c"))
500+
case Resolved(AuthTypes.OAUTH, Some(c @ Providers.CLIENT_CREDS), values)
501+
if !values.get(Keys.CLIENT_ENDPOINT).exists(clientCredsEndpointUsable) =>
502+
// The endpoint stays out of the reason: it may carry user info.
503+
Some(
504+
unsupported(
505+
s"auth type ${AuthTypes.OAUTH} with provider $c and an endpoint that is not an " +
506+
"https URL with a tenant path segment"))
499507
case Failed(_, message) =>
500508
// Some of Hadoop's own messages quote the whole URI, user info included.
501509
val safeMessage =
@@ -505,6 +513,23 @@ private[comet] object AbfsAuthResolver extends Logging {
505513
}
506514
}
507515

516+
/**
517+
* Whether the native store can post client credentials token requests to `endpoint`: an `https`
518+
* URL with a host and a tenant segment, which `client_creds_settings` in `azure.rs` reads the
519+
* same way (the tenant is the segment before the last `oauth2`, else the first). Conservative
520+
* where the native URL parser and `java.net.URI` could disagree: a percent escape, a backslash
521+
* or a dot segment is declined, since Hadoop's own reader may take it.
522+
*/
523+
private[objectstore] def clientCredsEndpointUsable(endpoint: String): Boolean =
524+
Try(new URI(endpoint)).toOption.exists { uri =>
525+
val path = Option(uri.getRawPath).getOrElse("")
526+
val segments = path.stripPrefix("/").split("/", -1)
527+
val tenantAt = Some(segments.lastIndexOf("oauth2")).filter(_ > 0).map(_ - 1).getOrElse(0)
528+
"https".equalsIgnoreCase(uri.getScheme) && uri.getHost != null &&
529+
!path.exists(c => c == '%' || c == '\\') && !segments.exists(s => s == "." || s == "..") &&
530+
segments(tenantAt).nonEmpty
531+
}
532+
508533
private def providerMarker(authType: String): String =
509534
if (authType == AuthTypes.SAS) Markers.SAS_PROVIDER_CLASS else Markers.OAUTH_PROVIDER_CLASS
510535

‎spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -342,7 +342,7 @@ object NativeConfig {
342342

343343
import scala.jdk.CollectionConverters._
344344
val options = scala.collection.mutable.Map[String, String]() ++= schemeListOptions(hadoopConf)
345-
val s3CompliantSchemes = parseSchemeSet(hadoopConf.get(COMET_S3_COMPLIANT_SCHEMES_KEY))
345+
val s3CompliantSchemes = resolveS3CompliantSchemes(hadoopConf)
346346

347347
// ABFS: Hadoop resolves the mechanism and values for this URI's account on the driver, and
348348
// only those travel to native (never another account's keys, never the whole namespace).

‎spark/src/test/scala/org/apache/comet/objectstore/NativeConfigSuite.scala‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -850,6 +850,68 @@ class NativeConfigSuite extends AnyFunSuite with Matchers {
850850
Providers.USER_PASSWORD)
851851
}
852852

853+
private def clientCredsConf(endpoint: String): Configuration =
854+
abfsScanConf(
855+
Keys.AUTH_TYPE -> "OAuth",
856+
Keys.OAUTH_PROVIDER_TYPE -> Providers.CLIENT_CREDS,
857+
Keys.CLIENT_ENDPOINT -> endpoint,
858+
Keys.CLIENT_ID -> "client-123",
859+
Keys.CLIENT_SECRET -> "client-secret-value")
860+
861+
test(
862+
"extractScanObjectStoreOptions - a client credentials endpoint native rejects falls back") {
863+
// `azure.rs` builds no store for an `http` endpoint or one without a tenant segment.
864+
Seq(
865+
"http://login.microsoftonline.com/tenant/oauth2/token",
866+
"https://login.microsoftonline.com/",
867+
"https://login.microsoftonline.com").foreach { endpoint =>
868+
withClue(s"$endpoint: ") {
869+
assertFallsBack(clientCredsConf(endpoint), Providers.CLIENT_CREDS, "tenant path segment")
870+
}
871+
}
872+
}
873+
874+
test("extractScanObjectStoreOptions - a client credentials endpoint password stays out") {
875+
val hadoopConf = clientCredsConf("http://user:[email protected]/t/oauth2/token")
876+
NativeConfig.extractScanObjectStoreOptions(
877+
hadoopConf,
878+
Seq(abfsFile("data", "accta", "a.parquet"))) match {
879+
case Left(reason) => reason should not include "endpoint-password"
880+
case Right(options) => fail(s"expected a fallback, got $options")
881+
}
882+
}
883+
884+
test("clientCredsEndpointUsable - agrees with the endpoints azure.rs builds a store for") {
885+
// The cases of `oauth_endpoint_parts_split_tenant_and_authority_host` in azure.rs, over https.
886+
val usable = Seq(
887+
"https://login.microsoftonline.com/tenant/oauth2/token",
888+
"https://user:[email protected]/t/oauth2/token",
889+
"https://auth-proxy.example:8443/t/oauth2/token",
890+
"https://[::1]:8443/t/oauth2/token",
891+
"https://auth-proxy.example/aad/t/oauth2/v2.0/token",
892+
"https://auth-proxy.example/aad/t/token",
893+
"https://auth-proxy.example/oauth2/token",
894+
"https://h//t/oauth2/token",
895+
"https://h/aad/t/oauth2/v2.0/token?q=oauth2#f",
896+
"https://h/a/oauth2/b/oauth2/token",
897+
"HTTPS://h/t/oauth2/token")
898+
val unusable = Seq(
899+
"http://login.microsoftonline.com/tenant/oauth2/token",
900+
"https://auth-proxy.example/",
901+
"https://auth-proxy.example",
902+
"https://h/a//oauth2/token",
903+
"file:///t/oauth2/token",
904+
"urn:t:oauth2",
905+
"not a url",
906+
// Resolved differently by the native URL parser, so declined rather than guessed at.
907+
"https://h/a/%2e%2e/t/oauth2/token",
908+
"https://h/a\\b/t/oauth2/token",
909+
"https://h/a/../oauth2/token")
910+
usable.foreach(e => withClue(e)(AbfsAuthResolver.clientCredsEndpointUsable(e) shouldBe true))
911+
unusable.foreach(e =>
912+
withClue(e)(AbfsAuthResolver.clientCredsEndpointUsable(e) shouldBe false))
913+
}
914+
853915
test("extractScanObjectStoreOptions - a failed ABFS resolution falls back to Spark") {
854916
assertFallsBack(
855917
abfsScanConf(Keys.AUTH_TYPE -> "NotAnAuthType"),

0 commit comments

Comments
 (0)