You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fd8bb59
Browse filesBrowse the repository at this point in the historyBrowse files
* fix: array expression audit follow-ups (#4503)
Address the actionable array expression audit follow-ups from #4503:
- ArrayReverse: report Incompatible for struct/map element types so they
route through the JVM codegen dispatcher and stay native, instead of
reporting Compatible while convert silently declined and fell back.
- SortArray: accept any foldable boolean ascendingOrder (Spark 4.0+ widens
it beyond a boolean Literal) and evaluate it at convert time.
- ArrayJoin: fall back for non-default string collations, mirroring
ArrayIntersect, so the limitation surfaces in EXPLAIN.
- ArrayExcept: surface the native element-type restriction in
getSupportLevel rather than only in convert.
- Remove the dead ArrayCompact serde registration (RuntimeReplaceable in
all supported versions, dispatched via ArrayFilter) and document
ArrayAppend as reachable only on Spark 3.x.
Item 2 (NaN/signed-zero gating for array_contains/distinct/union/max/min)
is intentionally omitted: existing SQL tests show native DataFusion already
canonicalizes NaN and signed zero to match Spark, so reporting Incompatible
would force a needless fallback to the codegen dispatcher with no
correctness benefit. Item 5 (array_union ordering) is tracked in #4681.
* refactor: simplify CometArrayExcept support-level and convert guards
Collapse the dual-Incompatible match in getSupportLevel into a single
Incompatible construction, and tighten the convert-time type guard to the
same find idiom (dropping a needless toSet). Behavior preserving.
* fix: keep collated array_join and array_intersect native via codegen dispatch
Address review feedback on #4713: report Incompatible instead of
Unsupported for non-default collations so both route through the JVM
codegen dispatcher (Spark's own doGenCode) and stay native while matching
Spark, reconciling the three #2190 collation cases with CometReverse.
array_join concatenation is collation-independent, so only the output
string's collation metadata is dropped. array_intersect membership is
collation-aware, so the dispatcher (not the byte-based native path)
produces the correct result. Add collation SQL tests for both.
0 commit comments