You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 14d2e7d
Browse filesBrowse the repository at this point in the historyBrowse files
refactor: move native -0.0 and NaN handling into one float_semantics module (apache#6400)
* refactor: move native -0.0 and NaN handling into one float_semantics module
Part of apache#6385. Each native expression that follows one of Spark's float
rules carried its own copy of that rule. This moves them onto a new
`float_semantics` module in `spark-expr`, with no change in behavior.
- `normalize_float`, `canonicalize_nan`, `compare_floats` and `hash_input`
are the per-value rules: `NormalizeNaNAndZero`, `Double.equals`,
`compareDoubles`, and the `Murmur3Hash`/`XxHash64` input.
- `normalize_floats`, `normalize_nested_floats` and `has_float_leaf`
normalize arrays. `NormalizeNaNAndZero` and `NormalizeNestedFloats`
move into the module, and `NormalizeNaNAndZero::wrap_if_needed` takes
over the planner's check in `create_normalized_key_expr`.
- `spark_comparator` replaces both `nested_equality` (nested `=` and
`IN`) and the comparator local to `array_min`/`array_max`.
hll_plus_plus, max_min_by, mode, percentile, array_extrema,
array_position, arrays_overlap, nested_comparison, the float hash macros,
and the planner's sort keys and range partition bounds now call the
module.
* Restore hot-loop speed and narrow the float_semantics API
- compare_floats uses the if-chain form again, so `.is_eq()` in
array_position and percentile compiles to a single comparison.
- float_extrema calls new float_lt/float_gt predicates, which compile to
the same instructions as the inline tests they replace. The three-way
comparison was up to 9x slower in that loop.
- spark_equality gives nested `=` and `IN` back the length check that
skips comparing the elements of lists of different lengths.
- float_semantics is a private module again, re-exporting
normalize_floats, canonicalize_nan and the two expressions.
- The Parquet variant cast canonicalizes NaN with canonicalize_nan, and
gains a test.
- Tests share the NaN constants and check the per-value rules in one
table, and the max_by doc defers to the module docs.
* Return an error, not a panic, for mismatched physical types in the comparator
`check_types` compares logical types, so a `Dictionary(_, Float64)` passes
as `Float64` and then panicked when `float_comparator` downcast it. The
comparator now matches on both sides' types and returns the internal error.
`mismatched_types_are_rejected` covers a dictionary on either side, at the
top level and inside a list.
0 commit comments