Repository navigation
Commit 88ec629
committed
fix: skip array_union's right operand on rows whose left array is NULL
Spark's ArrayUnion returns NULL for a NULL left array without evaluating
the right operand. The native function evaluates both over the whole
batch, so a right operand that throws, such as slice(b, 0, 1) or an ANSI
cast, failed on rows that Spark never evaluates it for. That already
happened on main for every element type that runs natively, and this PR
would have extended it to float elements on 4.0.5+ and 4.1.4+.
CometArrayUnion now guards the call with CASE WHEN <left> IS NOT NULL, as
CometElementAt does, unless the right operand is a column or a literal,
which cannot throw. A nullable nondeterministic left operand, which the
guard would evaluate twice, stays on Spark.1 parent 2ecb05f commit 88ec629
3 files changed
Lines changed: 147 additions & 4 deletions
File tree
- spark/src
- main/scala/org/apache/comet/serde
- test/resources/sql-tests/expressions/array
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
605 | 605 | | |
606 | 606 | | |
607 | 607 | | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
608 | 627 | | |
609 | 628 | | |
610 | 629 | | |
| 630 | + | |
| 631 | + | |
611 | 632 | | |
612 | | - | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
613 | 638 | | |
614 | 639 | | |
615 | 640 | | |
616 | 641 | | |
617 | 642 | | |
618 | | - | |
619 | | - | |
| 643 | + | |
| 644 | + | |
620 | 645 | | |
621 | 646 | | |
622 | 647 | | |
623 | 648 | | |
624 | 649 | | |
625 | 650 | | |
626 | | - | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
627 | 680 | | |
628 | 681 | | |
629 | 682 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
0 commit comments