You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use arrow make_comparator for nested structural equality in arrays_overlap and array_position #5101
Two array functions hand-roll nested element comparison that arrow's make_comparator provides:
native/spark-expr/src/array_funcs/arrays_overlap.rs (structural_eq / list_structural_eq / struct_structural_eq): recursive element-by-element structural equality for nested List/LargeList/FixedSizeList/Struct elements with null==null treated as equal, comparing leaves via a per-element one-row eq kernel call (a kernel dispatch per element pair).
native/spark-expr/src/array_funcs/array_position.rs (position_fallback): materializes a ScalarValue per candidate element inside the inner loop (an allocation per element) and compares with ScalarValue::eq.
Arrow's make_comparator (arrow-ord) supports fully nested types, treats null==null as Equal, and is built once per array pair, replacing both the recursion and the per-element dispatch/allocation.
Describe the potential solution
Build one comparator per (probe, search) array pair with SortOptions::default(), then compare indices with cmp(i, j) == Ordering::Equal per element.
Additional context
Float caveat: the current arrays_overlap leaf comparison uses the eq kernel (IEEE: NaN != NaN, -0.0 == 0.0), while make_comparator uses total order (NaN == NaN, -0.0 < 0.0). Spark's ordering.equiv (Java Double.compare) matches total order, so the comparator is arguably more Spark-correct, but it is a behavior change for NaN/-0.0 inside nested elements and needs test coverage before switching. For array_position, ScalarValue equality already matches total-order semantics, so that switch is behavior-preserving.
The typed fast paths in both files are not affected; there is no "search within list segments" kernel.
Found during an audit of native code that replicates existing arrow-rs kernels.
Activity
peterxcli commentedon Jul 31, 2026
take