You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Comparison operators on arrays and structs with floating-point leaves do not match Spark for signed zero #6157
<=>, <, <=, > and >= on arrays and structs with FLOAT/DOUBLE leaves run natively and compare the nested values with Arrow's total order, so -0.0 and 0.0 are different values. Spark compares nested values with TypeUtils.getInterpretedOrdering, which bottoms out in SQLOrderingUtil.compareDoubles/compareFloats, where signed zeros are equal.
CometExecRule.normalize only wraps scalar float operands. #6073 fixes =, <> and IN for nested operands, but leaves <=> and the ordering operators on the raw comparator. #5507 covers ORDER BY and rank on nested float keys, and this is the same gap in the comparison predicates.
Steps to reproduce
On main at cccc08b, default Spark 4.1 profile, as a CometSqlFileTestSuite fixture:
CREATETABLEt (a ARRAY<DOUBLE>, b ARRAY<DOUBLE>, s STRUCT<v: DOUBLE>, u STRUCT<v: DOUBLE>) USING parquet;
INSERT INTO t VALUES (
array(CAST('-0.0'AS DOUBLE)), array(CAST('0.0'AS DOUBLE)),
named_struct('v', CAST('-0.0'AS DOUBLE)), named_struct('v', CAST('0.0'AS DOUBLE)));
SELECT a <=> b FROM t;
SELECT a < b, a >= b, s <=> u, s < u FROM t;
Both queries run fully native (CometProject over CometNativeScan) and return different answers:
Query
Spark
Comet
a <=> b
true
false
a < b, a >= b, s <=> u, s < u
false, true, true, false
true, false, false, true
CAST('-0.0' AS DOUBLE) is a string cast, so the sign survives. CAST(-0.0 AS DOUBLE) would produce +0.0.
Expected behavior
The same answers as Spark: signed zeros inside arrays and structs compare equal for every comparison operator.
Additional context
Found while reviewing #6073. I only verified signed zero. NaN payloads and signs probably diverge too, since total_cmp orders a negative NaN below every other value while Spark treats every NaN as the largest value. The equivalence in #6073's nested_comparison.rs could likely be extended into an ordering comparator for these operators.
Activity
test: drop the sweep suite's known gaps that normalized comparisons c…
fix: normalize float operands in the native comparison builder (apach…