Repository navigation
Commit bf5dcbd
committed
[fix](be) Keep floating >= runtime filters out of the Lance scan SQL
### What problem does this PR solve?
Issue Number: None
Related PR: #69581
Problem Summary:
Reproduction: scan a Lance FLOAT or DOUBLE column that holds a negative NaN
(float bits 0xffc00000, double bits 0xfff8000000000000) under a runtime filter
`x >= c`, for example a MIN-only filter from a nested-loop join on `x >= y`.
Doris treats every NaN as the greatest value regardless of its sign
(`GreaterThanEqualsFloat` in `common/compare.h`), so the row satisfies the
filter, but it is missing from the result.
Root cause: `build_range_filter_sql()` translates the filter into the Lance
SQL `(x >= c)`. Lance evaluates it with Arrow/DataFusion comparison kernels
that order floating values by IEEE 754 totalOrder, where a negative NaN sorts
below every finite value, so Lance drops the row before the scanner evaluates
the Doris residual. The residual can only remove rows, not bring them back.
Fix: `build_range_filter_sql()` returns `std::nullopt` for a `>=` filter whose
slot is FLOAT or DOUBLE, so the filter stays residual-only and is reported
under `LanceRuntimeFilterSkippedIds`, the policy `PaimonRustPredicateConverter`
already applies to floating predicates. `<=` filters are still pushed: Lance
then only returns extra negative-NaN rows, which the residual removes. IN
filters are still pushed: a NaN row never matches a finite member, and sets
with NaN members were already rejected. A MIN_MAX filter now pushes only its
`<=` half.
Before: rows holding a negative NaN were dropped under a floating `>=` runtime
filter. After: Lance returns them and the Doris residual decides, so the
result matches a scan without the pushed filter.
branch-4.1 and branch-4.2 carry the identical helper, so the same change
applies there.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- New: `LanceRuntimeFilterHelperTest.KeepsFloatingMinFiltersResidualOnly`
(`>=` skipped, `<=` and IN still pushed, SQL of a MIN_MAX pair),
`LanceTableReaderFilterTest.PreservesNegativeNaNUnderFloatingMinRuntimeFilter`
(FLOAT and DOUBLE rows 0.5, 1, 2, +NaN, -NaN under `>= 1`; every row Doris
keeps must come back from Lance).
- Both pass on macOS arm64 (ASAN) with the Lance suites; with
`build_range_filter_sql()` as before this change, both fail.
- Behavior changed: No (the Lance reader is added by this PR; floating `>=`
runtime filters are no longer pushed into Lance, so such scans may read more
rows)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 05437fd commit bf5dcbd
3 files changed
Lines changed: 143 additions & 0 deletions
File tree
- be
- src/format_v2/lance
- test/format_v2
- lance
- table
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
403 | 403 | | |
404 | 404 | | |
405 | 405 | | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
406 | 419 | | |
407 | 420 | | |
408 | 421 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
272 | 272 | | |
273 | 273 | | |
274 | 274 | | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
275 | 319 | | |
276 | 320 | | |
277 | 321 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1967 | 1967 | | |
1968 | 1968 | | |
1969 | 1969 | | |
| 1970 | + | |
| 1971 | + | |
| 1972 | + | |
| 1973 | + | |
| 1974 | + | |
| 1975 | + | |
| 1976 | + | |
| 1977 | + | |
| 1978 | + | |
| 1979 | + | |
| 1980 | + | |
| 1981 | + | |
| 1982 | + | |
| 1983 | + | |
| 1984 | + | |
| 1985 | + | |
| 1986 | + | |
| 1987 | + | |
| 1988 | + | |
| 1989 | + | |
| 1990 | + | |
| 1991 | + | |
| 1992 | + | |
| 1993 | + | |
| 1994 | + | |
| 1995 | + | |
| 1996 | + | |
| 1997 | + | |
| 1998 | + | |
| 1999 | + | |
| 2000 | + | |
| 2001 | + | |
| 2002 | + | |
| 2003 | + | |
| 2004 | + | |
| 2005 | + | |
| 2006 | + | |
| 2007 | + | |
| 2008 | + | |
| 2009 | + | |
| 2010 | + | |
| 2011 | + | |
| 2012 | + | |
| 2013 | + | |
| 2014 | + | |
| 2015 | + | |
| 2016 | + | |
| 2017 | + | |
| 2018 | + | |
| 2019 | + | |
| 2020 | + | |
| 2021 | + | |
| 2022 | + | |
| 2023 | + | |
| 2024 | + | |
| 2025 | + | |
| 2026 | + | |
| 2027 | + | |
| 2028 | + | |
| 2029 | + | |
| 2030 | + | |
| 2031 | + | |
| 2032 | + | |
| 2033 | + | |
| 2034 | + | |
| 2035 | + | |
| 2036 | + | |
| 2037 | + | |
| 2038 | + | |
| 2039 | + | |
| 2040 | + | |
| 2041 | + | |
| 2042 | + | |
| 2043 | + | |
| 2044 | + | |
| 2045 | + | |
| 2046 | + | |
| 2047 | + | |
| 2048 | + | |
| 2049 | + | |
| 2050 | + | |
| 2051 | + | |
| 2052 | + | |
| 2053 | + | |
| 2054 | + | |
| 2055 | + | |
1970 | 2056 | | |
1971 | 2057 | | |
1972 | 2058 | | |
| |||
0 commit comments