Skip to content

Commit bf5dcbd

Browse files
committed
[fix](be) Keep floating >= runtime filters out of the Lance scan SQL
### What problem does this PR solve? Issue Number: None Related PR: #69581 Problem Summary: Reproduction: scan a Lance FLOAT or DOUBLE column that holds a negative NaN (float bits 0xffc00000, double bits 0xfff8000000000000) under a runtime filter `x >= c`, for example a MIN-only filter from a nested-loop join on `x >= y`. Doris treats every NaN as the greatest value regardless of its sign (`GreaterThanEqualsFloat` in `common/compare.h`), so the row satisfies the filter, but it is missing from the result. Root cause: `build_range_filter_sql()` translates the filter into the Lance SQL `(x >= c)`. Lance evaluates it with Arrow/DataFusion comparison kernels that order floating values by IEEE 754 totalOrder, where a negative NaN sorts below every finite value, so Lance drops the row before the scanner evaluates the Doris residual. The residual can only remove rows, not bring them back. Fix: `build_range_filter_sql()` returns `std::nullopt` for a `>=` filter whose slot is FLOAT or DOUBLE, so the filter stays residual-only and is reported under `LanceRuntimeFilterSkippedIds`, the policy `PaimonRustPredicateConverter` already applies to floating predicates. `<=` filters are still pushed: Lance then only returns extra negative-NaN rows, which the residual removes. IN filters are still pushed: a NaN row never matches a finite member, and sets with NaN members were already rejected. A MIN_MAX filter now pushes only its `<=` half. Before: rows holding a negative NaN were dropped under a floating `>=` runtime filter. After: Lance returns them and the Doris residual decides, so the result matches a scan without the pushed filter. branch-4.1 and branch-4.2 carry the identical helper, so the same change applies there. ### Release note None ### Check List (For Author) - Test: Unit Test - New: `LanceRuntimeFilterHelperTest.KeepsFloatingMinFiltersResidualOnly` (`>=` skipped, `<=` and IN still pushed, SQL of a MIN_MAX pair), `LanceTableReaderFilterTest.PreservesNegativeNaNUnderFloatingMinRuntimeFilter` (FLOAT and DOUBLE rows 0.5, 1, 2, +NaN, -NaN under `>= 1`; every row Doris keeps must come back from Lance). - Both pass on macOS arm64 (ASAN) with the Lance suites; with `build_range_filter_sql()` as before this change, both fail. - Behavior changed: No (the Lance reader is added by this PR; floating `>=` runtime filters are no longer pushed into Lance, so such scans may read more rows) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 05437fd commit bf5dcbd

3 files changed

Lines changed: 143 additions & 0 deletions

File tree

‎be/src/format_v2/lance/lance_runtime_filter_helper.cpp‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,19 @@ std::optional<std::string> build_range_filter_sql(const VExpr& predicate,
403403
// SQL.
404404
return std::nullopt;
405405
}
406+
const auto slot_type = remove_nullable(slot->data_type())->get_primitive_type();
407+
if (predicate.op() == TExprOpcode::GE &&
408+
(slot_type == TYPE_FLOAT || slot_type == TYPE_DOUBLE)) {
409+
// Doris orders every NaN above all other values regardless of its sign
410+
// (GreaterThanEqualsFloat in common/compare.h), so `x >= 1` keeps NaN rows. Lance
411+
// evaluates the pushed SQL with Arrow/DataFusion kernels that use IEEE 754 totalOrder,
412+
// where a negative NaN such as 0xffc00000 sorts below every finite value: `x >= 1` drops
413+
// that row before the Doris residual can keep it. A MIN-only filter, for example from a
414+
// nested-loop join on `x >= y`, would lose such rows. `<=` only lets negative NaNs
415+
// through to the residual, which removes them, and IN sets with NaN members are not
416+
// pushed. PaimonRustPredicateConverter keeps floating predicates residual-only too.
417+
return std::nullopt;
418+
}
406419
const auto physical_type = find_unique_physical_type(physical_schema, slot->column_name());
407420
if (physical_type == nullptr) {
408421
// A missing or ambiguous top-level field cannot be bound safely by column name.

‎be/test/format_v2/lance/lance_runtime_filter_helper_test.cpp‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,50 @@ TEST(LanceRuntimeFilterHelperTest, SkipsPhysicalNumericTypesUnsupportedByPinnedP
272272
EXPECT_EQ((std::vector<int> {25, 26, 27, 28}), result->skipped_filter_ids);
273273
}
274274

275+
TEST(LanceRuntimeFilterHelperTest, KeepsFloatingMinFiltersResidualOnly) {
276+
const auto schema = arrow::schema(
277+
{arrow::field("f32", arrow::float32()), arrow::field("f64", arrow::float64())});
278+
279+
// Lance's totalOrder sorts a negative NaN below 1, while Doris keeps it for `x >= 1`.
280+
const VExprContextSPtrs min_filters {
281+
typed_runtime_range<TYPE_FLOAT>("f32", TExprOpcode::GE, 1.0F,
282+
std::make_shared<DataTypeFloat32>(), 31),
283+
typed_runtime_range<TYPE_DOUBLE>("f64", TExprOpcode::GE, 1.0,
284+
std::make_shared<DataTypeFloat64>(), 32),
285+
};
286+
const auto skipped = get_or_create_lance_runtime_filter_sql(min_filters, *schema, nullptr);
287+
ASSERT_NE(skipped, nullptr);
288+
EXPECT_TRUE(skipped->expression.empty());
289+
EXPECT_TRUE(skipped->pushable_filter_ids.empty());
290+
EXPECT_EQ((std::vector<int> {31, 32}), skipped->skipped_filter_ids);
291+
292+
// `<=` can only let negative NaNs through to the residual, and IN never matches a NaN row.
293+
const VExprContextSPtrs max_and_in_filters {
294+
typed_runtime_range<TYPE_FLOAT>("f32", TExprOpcode::LE, 1.0F,
295+
std::make_shared<DataTypeFloat32>(), 33),
296+
typed_runtime_in<TYPE_FLOAT>("f32", 1.5F, std::make_shared<DataTypeFloat32>(), 34),
297+
};
298+
const auto pushed =
299+
get_or_create_lance_runtime_filter_sql(max_and_in_filters, *schema, nullptr);
300+
ASSERT_NE(pushed, nullptr);
301+
EXPECT_EQ("(`f32` <= 1) AND (`f32` IN (1.5))", pushed->expression);
302+
EXPECT_EQ((std::vector<int> {33, 34}), pushed->pushable_filter_ids);
303+
EXPECT_TRUE(pushed->skipped_filter_ids.empty());
304+
305+
// The two halves of a MIN_MAX filter share one id; only its `<=` half reaches Lance.
306+
const VExprContextSPtrs min_max_filter {
307+
typed_runtime_range<TYPE_DOUBLE>("f64", TExprOpcode::GE, 1.0,
308+
std::make_shared<DataTypeFloat64>(), 35),
309+
typed_runtime_range<TYPE_DOUBLE>("f64", TExprOpcode::LE, 2.0,
310+
std::make_shared<DataTypeFloat64>(), 35),
311+
};
312+
const auto min_max = get_or_create_lance_runtime_filter_sql(min_max_filter, *schema, nullptr);
313+
ASSERT_NE(min_max, nullptr);
314+
EXPECT_EQ("(`f64` <= 2)", min_max->expression);
315+
EXPECT_EQ((std::vector<int> {35}), min_max->pushable_filter_ids);
316+
EXPECT_TRUE(min_max->skipped_filter_ids.empty());
317+
}
318+
275319
TEST(LanceRuntimeFilterHelperTest, ReusesSnapshotAcrossParallelReaders) {
276320
ShardedKVCache cache(2);
277321
const VExprContextSPtrs first_conjuncts {

‎be/test/format_v2/table/lance_reader_test.cpp‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1967,6 +1967,92 @@ TEST(LanceTableReaderFilterTest, SkipsTimestampNanoRuntimeFilterBeforeMaterializ
19671967
EXPECT_TRUE(reader.close().ok());
19681968
}
19691969

1970+
TEST(LanceTableReaderFilterTest, PreservesNegativeNaNUnderFloatingMinRuntimeFilter) {
1971+
const auto unique_suffix = std::chrono::steady_clock::now().time_since_epoch().count();
1972+
const auto dataset_uri =
1973+
std::filesystem::temp_directory_path() /
1974+
("doris_lance_negative_nan_rf_" + std::to_string(unique_suffix) + ".lance");
1975+
Defer cleanup {[&] {
1976+
std::error_code error;
1977+
std::filesystem::remove_all(dataset_uri, error);
1978+
}};
1979+
1980+
// Rows 1..5 hold 0.5, 1, 2, +NaN, and a negative NaN in both columns.
1981+
const auto negative_nan_f32 = std::bit_cast<float>(uint32_t {0xffc00000});
1982+
const auto negative_nan_f64 = std::bit_cast<double>(uint64_t {0xfff8000000000000});
1983+
ASSERT_TRUE(std::isnan(negative_nan_f32) && std::signbit(negative_nan_f32));
1984+
ASSERT_TRUE(std::isnan(negative_nan_f64) && std::signbit(negative_nan_f64));
1985+
arrow::Int64Builder row_id_builder;
1986+
ASSERT_TRUE(row_id_builder.AppendValues({1, 2, 3, 4, 5}).ok());
1987+
std::shared_ptr<arrow::Array> row_ids;
1988+
ASSERT_TRUE(row_id_builder.Finish(&row_ids).ok());
1989+
arrow::FloatBuilder f32_builder;
1990+
ASSERT_TRUE(f32_builder
1991+
.AppendValues({0.5F, 1.0F, 2.0F, std::numeric_limits<float>::quiet_NaN(),
1992+
negative_nan_f32})
1993+
.ok());
1994+
std::shared_ptr<arrow::Array> f32_values;
1995+
ASSERT_TRUE(f32_builder.Finish(&f32_values).ok());
1996+
arrow::DoubleBuilder f64_builder;
1997+
ASSERT_TRUE(f64_builder
1998+
.AppendValues({0.5, 1.0, 2.0, std::numeric_limits<double>::quiet_NaN(),
1999+
negative_nan_f64})
2000+
.ok());
2001+
std::shared_ptr<arrow::Array> f64_values;
2002+
ASSERT_TRUE(f64_builder.Finish(&f64_values).ok());
2003+
const auto schema = arrow::schema({arrow::field("row_id", arrow::int64()),
2004+
arrow::field("f32", arrow::float32()),
2005+
arrow::field("f64", arrow::float64())});
2006+
ASSERT_TRUE(write_lance_record_batch(
2007+
dataset_uri,
2008+
arrow::RecordBatch::Make(schema, 5, {row_ids, f32_values, f64_values}))
2009+
.ok());
2010+
2011+
const Columns columns {projected_column("row_id", TYPE_BIGINT, false)};
2012+
for (const bool use_double : {false, true}) {
2013+
SCOPED_TRACE(use_double ? "f64" : "f32");
2014+
const auto runtime_filter = use_double ? create_runtime_range_conjunct<TYPE_DOUBLE>(
2015+
"f64", TExprOpcode::GE, 1.0,
2016+
std::make_shared<DataTypeFloat64>(), 52)
2017+
: create_runtime_range_conjunct<TYPE_FLOAT>(
2018+
"f32", TExprOpcode::GE, 1.0F,
2019+
std::make_shared<DataTypeFloat32>(), 51);
2020+
TQueryGlobals query_globals;
2021+
RuntimeState state(TQueryOptions(), query_globals);
2022+
RuntimeProfile profile("lance_negative_nan_runtime_filter_fixture");
2023+
TFileScanRangeParams scan_params;
2024+
LanceTableReader reader;
2025+
ASSERT_TRUE(init_reader(&reader, columns, &state, &profile, &scan_params, {runtime_filter})
2026+
.ok());
2027+
ASSERT_TRUE(prepare_range(&reader, make_latest_lance_range(dataset_uri)).ok());
2028+
2029+
Block block;
2030+
add_output_columns(&block, columns);
2031+
std::vector<int64_t> returned_row_ids;
2032+
bool eos = false;
2033+
while (!eos) {
2034+
ASSERT_TRUE(reader.get_block(&block, &eos).ok());
2035+
if (!eos) {
2036+
const auto& values =
2037+
assert_cast<const ColumnInt64&>(*block.get_by_position(0).column);
2038+
returned_row_ids.insert(returned_row_ids.end(), values.get_data().begin(),
2039+
values.get_data().end());
2040+
}
2041+
}
2042+
EXPECT_TRUE(reader.close().ok());
2043+
const auto* skipped_ids = profile.get_info_string("LanceRuntimeFilterSkippedIds");
2044+
ASSERT_NE(nullptr, skipped_ids);
2045+
EXPECT_EQ(use_double ? "52" : "51", *skipped_ids);
2046+
// Doris keeps rows 2..5 for `x >= 1`, both NaNs included. A pushed `x >= 1` let Lance drop
2047+
// row 5, which sorts below every finite value in totalOrder. Row 1 may come back too: the
2048+
// Doris residual removes it.
2049+
std::ranges::sort(returned_row_ids);
2050+
for (const int64_t row_id : {2, 3, 4, 5}) {
2051+
EXPECT_TRUE(std::ranges::binary_search(returned_row_ids, row_id)) << row_id;
2052+
}
2053+
}
2054+
}
2055+
19702056
TEST(LanceTableReaderFilterTest, SkipsPhysicalNumericTypesUnsupportedByPinnedPlanner) {
19712057
const auto unique_suffix = std::chrono::steady_clock::now().time_since_epoch().count();
19722058
const auto dataset_uri =

0 commit comments

Comments
 (0)