Repository navigation
[core] Keep FLOAT/DOUBLE files whose NaN rows are left out of stats - #1160
JeonDaehong wants to merge 1 commit into
Conversation
|
@JingsongLi @leaves12138 could you take a look when you have time? Small fix in stats pruning, same kind as #926. |
|
The declared positive-NaN fix has end-to-end value. Local validation passed 93 checks across statistics, Arrow residuals, PK value statistics, predicate deletes, the added native Parquet test, and an additional FLOAT32 version. The positive-NaN Parquet regression fails on base 46e6871 and passes here for both FLOAT and DOUBLE; I found no introduced correctness regression in the diff. A separate pre-existing boundary surfaced during production-path validation: negative NaNs do not currently behave like Java's canonical NaN in the Arrow residual. With FLOAT |
The writer leaves NaN out of FLOAT/DOUBLE min/max without counting it null, and NaN orders above every number in Java and in the row filter. So a file holding [0.0, NaN] reports max == 0.0, and f > 1.0, f >= 1.0, NOT BETWEEN and the negation of <, <= and BETWEEN pruned it, dropping the NaN row. These now keep FLOAT/DOUBLE files, as the NotEq/NotIn arms already do (apache#926).
bb7c78d to
665ce3d
Compare
|
Thanks for checking this. I confirmed it with a Java probe: In this PR I narrowed the comments to say "Java's ordering" (665ce3d), with no code change. I opened #1174 for the residual and will send a PR for it. |
|
Closing along with #1177, see the note there. |
Purpose
Linked issue: close #1159
Brief change log
The writer leaves NaN out of FLOAT/DOUBLE min/max, so
maxcan't be trusted to rule out a NaN row.>,>=,NOT BETWEEN, andNOTof<,<=,BETWEENnow keep FLOAT/DOUBLE files instead of pruning them bymax. Same idea as #926 did forNOT(=).The cost is that FLOAT/DOUBLE columns no longer prune on these operators. Other types are unchanged.
Tests
range_predicates_on_float_keep_files_that_may_hold_nan(unit)float_nan_stats_pruning_test: files[0.0, NaN]and[2.0],f > 1.0returned[2]before,[1, 2]nowequality_exclusion_falls_open_for_floats(fix(scan): prune NOT IN predicates using file stats #795) expectedGtto keep pruning a FLOAT file. Changed that line;Ltstill prunes.Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).
API and Format
None.
Documentation
None.