Repository navigation
Commit 61fbbe2
authored
fix: fall back to Spark when the driver cannot use the ABFS authentication (#7221)
* fix: fall back to Spark when the driver cannot use the ABFS authentication
A native ABFS scan whose authentication Hadoop declines, resolves to a
RefreshToken or UserPassword OAuth provider, or fails to resolve on the
driver now falls back to Spark's own scan instead of planning natively
and failing every task.
Closes #7202
* fix: keep a URL password out of the ABFS fallback reason
* fix: keep a URL password out of Hadoop messages in the ABFS fallback reason
Strip the password from the user info wherever a Failed message quotes it,
cover the supported OAuth and SAS mechanisms staying native, and add
end-to-end fallback tests for Parquet and CSV scans.
* fix: fall back for a client credentials endpoint the native ABFS store rejects
nativeFallbackReason now declines a Resolved ClientCredsTokenProvider whose
endpoint is not https or has no tenant segment, which azure.rs rejects, so the
scan reads through Spark instead of failing every task. The reason leaves the
endpoint out because it can carry user info.
Also reuse resolveS3CompliantSchemes in objectStoreOptionsAndAliasOptions.
* fix: fall back for an http Workload Identity authority in the native ABFS scan
object_store's token client refuses http, so a Resolved WorkloadIdentityTokenProvider
whose fs.azure.account.oauth2.msi.authority is not https plans natively and fails
every task, while Hadoop posts to it. nativeFallbackReason now declines it. The
reason leaves the authority out because it can carry user info.
* fix: ask native whether it can build the ABFS store instead of copying its rules
The driver now calls NativeBase.abfsStoreError, which runs azure::create_store on the
resolved options, and falls back with native's own message when it fails. The Scala
copies of the client credentials endpoint and Workload Identity authority rules are
removed. Native rejects an http endpoint or authority when it builds the store, so
native writes get that error up front too.
* fix: fall back when asking native about the ABFS store fails, and type the probe's arrays
Only a LinkageError (the library is not loaded) assumes the store builds. Any other failure from the probe, such as a panic while building the store, would fail every task of a native scan, so it is now a fallback reason. The JNI function takes JObjectArray<JString>, which drops the unsafe JString casts.1 parent 967d09b commit 61fbbe2
8 files changed
Lines changed: 512 additions & 26 deletions
File tree
- docs/source/user-guide/latest
- native/core/src
- parquet/objectstore
- spark/src
- main
- java/org/apache/comet
- scala/org/apache/comet/objectstore
- test/scala/org/apache/comet/objectstore
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
462 | 462 | | |
463 | 463 | | |
464 | 464 | | |
465 | | - | |
| 465 | + | |
466 | 466 | | |
467 | 467 | | |
468 | 468 | | |
| |||
479 | 479 | | |
480 | 480 | | |
481 | 481 | | |
482 | | - | |
| 482 | + | |
483 | 483 | | |
484 | | - | |
| 484 | + | |
485 | 485 | | |
486 | 486 | | |
487 | 487 | | |
488 | 488 | | |
489 | 489 | | |
490 | | - | |
| 490 | + | |
491 | 491 | | |
492 | 492 | | |
493 | 493 | | |
| |||
557 | 557 | | |
558 | 558 | | |
559 | 559 | | |
560 | | - | |
| 560 | + | |
561 | 561 | | |
562 | 562 | | |
563 | 563 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
| |||
319 | 320 | | |
320 | 321 | | |
321 | 322 | | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
322 | 350 | | |
323 | 351 | | |
324 | 352 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
40 | 45 | | |
41 | 46 | | |
42 | 47 | | |
| |||
62 | 67 | | |
63 | 68 | | |
64 | 69 | | |
65 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
66 | 73 | | |
67 | 74 | | |
68 | 75 | | |
| |||
129 | 136 | | |
130 | 137 | | |
131 | 138 | | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
132 | 154 | | |
133 | 155 | | |
134 | 156 | | |
| |||
232 | 254 | | |
233 | 255 | | |
234 | 256 | | |
| 257 | + | |
235 | 258 | | |
236 | 259 | | |
237 | 260 | | |
| |||
274 | 297 | | |
275 | 298 | | |
276 | 299 | | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
277 | 308 | | |
278 | 309 | | |
279 | 310 | | |
| |||
287 | 318 | | |
288 | 319 | | |
289 | 320 | | |
290 | | - | |
291 | | - | |
292 | | - | |
293 | | - | |
| 321 | + | |
294 | 322 | | |
295 | 323 | | |
296 | 324 | | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
297 | 338 | | |
298 | 339 | | |
299 | 340 | | |
| |||
529 | 570 | | |
530 | 571 | | |
531 | 572 | | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
532 | 600 | | |
533 | 601 | | |
534 | 602 | | |
| |||
563 | 631 | | |
564 | 632 | | |
565 | 633 | | |
| 634 | + | |
566 | 635 | | |
567 | 636 | | |
568 | 637 | | |
| 638 | + | |
569 | 639 | | |
570 | 640 | | |
571 | 641 | | |
| |||
694 | 764 | | |
695 | 765 | | |
696 | 766 | | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
697 | 776 | | |
698 | 777 | | |
699 | 778 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
401 | 401 | | |
402 | 402 | | |
403 | 403 | | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
404 | 417 | | |
0 commit comments