fix: stop constant-folding float columns whose equal bounds hide NaN by Shahzeb-DataEngineer · Pull Request #5012 · delta-io/delta-rs

Shahzeb-DataEngineer · 2026-11-02T05:15:50Z

Description

A file holding [NaN, 9.0] in a float column was recorded in the Delta log
with min = max = 9.0 and 0 nulls, because Parquet statistics leave NaN out
of float min/max. DataFusion constant-folds a column when its file
statistics show equal Exact bounds with an Exact(0) null count, so scans
returned 9.0 for the NaN row, and writes built on those scans (update,
delete, merge, overwrite with a predicate) made the replacement permanent.

The scan statistics built in replay.rs now downgrade the null count to
inexact for float columns whose min and max bounds are equal. That breaks
exactly the combination DataFusion constant-folds from, while leaving the
bounds Exact, so file pruning by min/max still works. Only equal-bound
float files lose an exact null count (null-predicate pruning for those
files); every other column and every unequal-bound float column is
unchanged.

Related Issue

Closes #5007

Testing

New unit tests test_adjust_float_null_count_downgrades_equal_float_bounds
and test_adjust_float_null_count_keeps_other_cases_exact in
table_provider/next/scan/replay.rs.
cargo test -p deltalake-core --features datafusion --lib delta_datafusion::table_provider::next::scan::replay: 4 passed, 0
failed (the 2 new tests plus the 2 pre-existing replay tests).
cargo clippy -p deltalake-core --features datafusion --lib: clean.
The touched lines are rustfmt-clean; the file's pre-existing import
blocks differ from this machine's rustfmt version and were left alone.

Note: an end-to-end check through the Python bindings was not run here;
the fix is at the statistics layer, and the unit tests cover the changed
combination. The same constant-fold path in datafusion-datasource-parquet
(apache/datafusion#26494) is upstream work, not part of this change.

AI assistance disclosure

This change was prepared with AI assistance; the design and every line
were reviewed against the source, and the tests above were run locally
before submission.

Parquet statistics leave NaN out of float min/max, so a file holding [NaN, 9.0] is recorded with min = max = 9.0 and 0 nulls. DataFusion constant-folds a column from equal Exact bounds with an Exact(0) null count, silently rewriting NaN rows on reads and on write-back. Report the null count as inexact for equal-bound float columns so the file is read row by row, while keeping the bounds Exact so file pruning still works. Closes delta-io#5007 Signed-off-by: Shahzeb Ali <[email protected]>

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

ion-elgreco · 2026-11-09T23:11:53Z

can you create a python repro that would trigger this bug?

Regression test for delta-io#5007: a file holding [NaN, 9.0] records equal float min/max bounds, which DataFusion constant-folded, so the NaN row was returned (and overwritten or deleted) as 9.0. Covers the read, update and delete paths from the issue repro. Signed-off-by: Shahzeb Ali <[email protected]>

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Shahzeb-DataEngineer · 2026-11-11T10:25:45Z

Thanks for the suggestion. Added a Python regression test in python/tests/test_query.py (test_query_builder_nan_float_with_equal_min_max_not_constant_folded) covering the read, update and delete paths from #5007. Verified it fails on the released deltalake 1.6.6 build, which does not have this fix: WHERE f = 9.0 returns [(1, 9.0), (2, 9.0)] instead of [(2, 9.0)], and the update and delete paths also treat the NaN row as 9.0. It should pass with this fix once CI runs it against the PR build.

ion-elgreco · 2026-11-11T11:43:22Z

+                        // Parquet statistics leave NaN out of float min/max, so equal
+                        // Exact bounds plus an Exact(0) null count do not prove every
+                        // row holds that value, yet that is exactly the combination
+                        // DataFusion constant-folds a column from: NaN rows were
+                        // rewritten to the bound value (#5007). Downgrade the null
+                        // count for equal-bound float columns so the file is read row
+                        // by row; bound-based pruning still applies.
+                        let null_count =
+                            adjust_float_null_count(&min_value, &max_value, null_count);


AI code suggestion

Suggested change

// Parquet statistics leave NaN out of float min/max, so equal

// Exact bounds plus an Exact(0) null count do not prove every

// row holds that value, yet that is exactly the combination

// DataFusion constant-folds a column from: NaN rows were

// rewritten to the bound value (#5007). Downgrade the null

// count for equal-bound float columns so the file is read row

// by row; bound-based pruning still applies.

let null_count =

adjust_float_null_count(&min_value, &max_value, null_count);

let null_count =

adjust_float_null_count(&min_value, &max_value, null_count);

ion-elgreco · 2026-11-11T11:43:22Z

+/// Downgrade the null count for a float column whose min and max bounds are
+/// equal.
+///
+/// Parquet statistics exclude NaN from float min/max, so a file holding
+/// `[NaN, 9.0]` is recorded with min = max = 9.0 and 0 nulls. Together with
+/// an Exact null count, DataFusion treats that combination as proof the
+/// column is constant and replaces it with the bound value, silently
+/// rewriting NaN rows on reads and on write-back (#5007). Reporting the
+/// null count as inexact breaks exactly that combination while leaving the
+/// bounds Exact, so file pruning by min/max still works.
+fn adjust_float_null_count(
+    min_value: &Precision<ScalarValue>,
+    max_value: &Precision<ScalarValue>,
+    null_count: Precision<usize>,
+) -> Precision<usize> {
+    let equal_bounds = matches!(
+        (min_value, max_value),
+        (Precision::Exact(min), Precision::Exact(max)) if min == max && !min.is_null()
+    );
+    let is_float = matches!(
+        min_value,
+        Precision::Exact(ScalarValue::Float32(_)) | Precision::Exact(ScalarValue::Float64(_))
+    );
+    if is_float && equal_bounds {
+        match null_count {
+            Precision::Exact(n) => Precision::Inexact(n),
+            other => other,
+        }
+    } else {
+        null_count
+    }
+}
+


AI code suggestion

Suggested change

/// Downgrade the null count for a float column whose min and max bounds are

/// equal.

///

/// Parquet statistics exclude NaN from float min/max, so a file holding

/// `[NaN, 9.0]` is recorded with min = max = 9.0 and 0 nulls. Together with

/// an Exact null count, DataFusion treats that combination as proof the

/// column is constant and replaces it with the bound value, silently

/// rewriting NaN rows on reads and on write-back (#5007). Reporting the

/// null count as inexact breaks exactly that combination while leaving the

/// bounds Exact, so file pruning by min/max still works.

fn adjust_float_null_count(

min_value: &Precision<ScalarValue>,

max_value: &Precision<ScalarValue>,

null_count: Precision<usize>,

) -> Precision<usize> {

let equal_bounds = matches!(

(min_value, max_value),

(Precision::Exact(min), Precision::Exact(max)) if min == max && !min.is_null()

);

let is_float = matches!(

min_value,

Precision::Exact(ScalarValue::Float32(_)) | Precision::Exact(ScalarValue::Float64(_))

);

if is_float && equal_bounds {

match null_count {

Precision::Exact(n) => Precision::Inexact(n),

other => other,

}

} else {

null_count

}

}

/// Downgrade the null count of a float column whose min and max are equal.

///

/// Parquet leaves NaN out of float min/max, so equal bounds do not prove a

/// constant column. An inexact null count stops DataFusion from

/// constant-folding it (#5007); min/max pruning still applies.

// TODO: remove when apache/datafusion#26490 is fixed.

fn adjust_float_null_count(

min_value: &Precision<ScalarValue>,

max_value: &Precision<ScalarValue>,

null_count: Precision<usize>,

) -> Precision<usize> {

match (min_value, max_value) {

(

Precision::Exact(min @ (ScalarValue::Float32(Some(_)) | ScalarValue::Float64(Some(_)))),

Precision::Exact(max),

) if min == max => null_count.to_inexact(),

_ => null_count,

}

}

ion-elgreco · 2026-11-11T11:43:22Z

+    use datafusion::{common::stats::Precision, scalar::ScalarValue};
+
+    use super::{adjust_float_null_count, visit_scan_file, ScanContext};


AI code suggestion

Suggested change

use datafusion::{common::stats::Precision, scalar::ScalarValue};

use super::{adjust_float_null_count, visit_scan_file, ScanContext};

use datafusion::{common::stats::Precision, scalar::ScalarValue};

use super::{ScanContext, adjust_float_null_count, visit_scan_file};

Copilot AI balanced review requested due to automatic review settings November 2, 2026 05:15

github-project-automation Bot added this to delta-rust Nov 2, 2026

Copilot AI reviewed Nov 2, 2026

Shahzeb-DataEngineer requested a balanced review from Copilot November 2, 2026 05:16

Shahzeb-DataEngineer requested review from ethan-tyler, ion-elgreco, roeap and rtyler as code owners November 2, 2026 05:16

Copilot AI reviewed Nov 2, 2026

github-actions Bot added the binding/rust Issues for the Rust crate label Nov 2, 2026

ion-elgreco self-assigned this Nov 9, 2026

Copilot AI balanced review requested due to automatic review settings November 10, 2026 08:04

Copilot AI reviewed Nov 10, 2026

github-actions Bot added the binding/python Issues for the Python package label Nov 10, 2026

ion-elgreco reviewed Nov 11, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix: stop constant-folding float columns whose equal bounds hide NaN - #5012

fix: stop constant-folding float columns whose equal bounds hide NaN#5012
Shahzeb-DataEngineer wants to merge 2 commits into
delta-io:mainfrom
Shahzeb-DataEngineer:fix/float-nan-constant-fold

Shahzeb-DataEngineer commented Nov 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

ion-elgreco commented Nov 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Shahzeb-DataEngineer commented Nov 11, 2026

Uh oh!

ion-elgreco Nov 11, 2026 •

edited

Loading

Uh oh!

ion-elgreco Nov 11, 2026 •

edited

Loading

Uh oh!

ion-elgreco Nov 11, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		use datafusion::{common::stats::Precision, scalar::ScalarValue};

		use super::{adjust_float_null_count, visit_scan_file, ScanContext};

Conversation

Shahzeb-DataEngineer commented Nov 2, 2026

Description

Related Issue

Testing

AI assistance disclosure

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

ion-elgreco commented Nov 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Shahzeb-DataEngineer commented Nov 11, 2026

Uh oh!

ion-elgreco Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ion-elgreco Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ion-elgreco Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ion-elgreco Nov 11, 2026 •

edited

Loading

ion-elgreco Nov 11, 2026 •

edited

Loading

ion-elgreco Nov 11, 2026 •

edited

Loading