Repository navigation
Commit 881b871
committed
fix: report map builder errors in Spark's row order and trim the per-batch validation
The map_from_arrays validator checked every row's key and value lengths
before the kernel looked for a duplicate key, so a duplicate in row 0
followed by a length mismatch in row 1 raised _LEGACY_ERROR_TEMP_2128
where Spark raises DUPLICATED_MAP_KEY. The kernel already checks lengths
and duplicates row by row, in Spark's order, so both validators now
return early unless a row that gets built holds a NULL key; that row is
bound to fail, so only then are the rows up to it walked, lengths first.
The kernel's length-mismatch message is translated to
SparkError::MapKeyValueDiffSizes like its duplicate-key message, with a
pin test. map_from_entries first rules out the common case with one AND
and a popcount (a NULL key only under NULL entries), and the walk reuses
the duplicate from HashSet::replace instead of cloning every key.
The sliced-list workaround now rebases only a list whose first offset
is nonzero, zero-copy (shifted offsets and a values slice), links
apache/datafusion#25419, and says to delete it once Comet picks up
apache/datafusion#25431. An all-scalar call is evaluated once and
returned as a scalar, as the kernel alone does, instead of building
number_rows rows.
Cleanups: derive Default for the wrappers and drop their unused new(),
read key nulls with logical_nulls(), drop the unreachable LargeList and
FixedSizeList arms and row-count check, drop the clippy allow (clippy is
clean without it), parse the policy into the typed ConfigOptions field
instead of set_str, and inline session_config_options() at its only
caller. The Rust tests that only exercised the upstream kernel are gone;
new ones pin the orderings above, a NULL key in a row whose values are
NULL or that holds a NULL entry, and the all-scalar result.1 parent 88c877b commit 881b871
3 files changed
Lines changed: 416 additions & 525 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
759 | 759 | | |
760 | 760 | | |
761 | 761 | | |
762 | | - | |
763 | | - | |
764 | | - | |
765 | | - | |
766 | | - | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
767 | 766 | | |
768 | | - | |
| 767 | + | |
769 | 768 | | |
770 | 769 | | |
771 | 770 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3666 | 3666 | | |
3667 | 3667 | | |
3668 | 3668 | | |
3669 | | - | |
3670 | | - | |
3671 | | - | |
3672 | | - | |
3673 | | - | |
3674 | | - | |
3675 | | - | |
3676 | | - | |
3677 | 3669 | | |
3678 | 3670 | | |
3679 | 3671 | | |
| |||
3800 | 3792 | | |
3801 | 3793 | | |
3802 | 3794 | | |
3803 | | - | |
| 3795 | + | |
| 3796 | + | |
| 3797 | + | |
| 3798 | + | |
3804 | 3799 | | |
3805 | 3800 | | |
3806 | 3801 | | |
| |||
0 commit comments