Plain SELECT of a Parquet float column returns wrong values: NaN rows are replaced by the statistics constant when min == max

Describe the bug

Parquet writers leave NaN out of floating-point min/max statistics (the spec requires it). So a column chunk holding
one distinct finite value plus some NaNs has statistics min == max == x and null_count == 0, although its values are
not constant. DataFusion treats such a column as constant and substitutes the statistic for the data. A plain
SELECT * FROM 'file.parquet' then returns x for every NaN row. There is no error, and NaNs silently disappear.
pyarrow, DuckDB and Polars read the same file correctly.

This is not limited to pruning: datafusion.execution.parquet.pruning = false and enable_page_index = false do not
change the result. datafusion.execution.collect_statistics = false does. A COPY … TO from such a file writes the
wrong values into the new files.

The shortcut is constant_columns_from_stats / constant_value_from_stats in
datafusion/datasource-parquet/src/opener/mod.rs (lines 1968 and 1999, called at line 891): min == max && !min.is_null() && null_count == Exact(0). For Float16/32/64, Parquet min/max cannot prove the column is constant, because NaN is excluded from them.
So floating-point columns should not be treated as constant from these statistics (unless a NaN count of 0 is known).

Draft PR #24804 (for #15812, pruning with NaN) would disable floating-point min/max in a way that also covers this
("constant-column substitution"), but it is a draft and has no issue for this user-visible symptom. I'm filing it so
the wrong-result case is tracked and tested.

To Reproduce

import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({'c0': pa.array([float('nan'), -8.6e-27])}), 'nan.parquet')
pq.write_table(pa.table({'c0': pa.array([1.5, float('nan')], pa.float32())}), 'nan_f32.parquet')
pq.write_table(pa.table({'c0': pa.array([float('nan'), -8.6e-27])}), 'nan_nostats.parquet', write_statistics=False)

$ datafusion-cli -q -c "SELECT * FROM 'nan.parquet';" -c "SELECT count(*) FILTER (WHERE isnan(c0)) AS nans FROM 'nan.parquet';"
+----------+
| c0       |
+----------+
| -8.6e-27 |
| -8.6e-27 |      <-- this row is NaN in the file
+----------+
+------+
| nans |
+------+
| 0    |
+------+

$ datafusion-cli -q -c "SELECT * FROM 'nan_f32.parquet';"
| 1.5 |
| 1.5 |            <-- NaN in the file

$ datafusion-cli -q -c "SELECT * FROM 'nan_nostats.parquet';"
| NaN      |
| -8.6e-27 |       <-- correct without statistics

$ datafusion-cli -q -c "SET datafusion.execution.collect_statistics = false;" -c "SELECT * FROM 'nan.parquet';"
| NaN      |
| -8.6e-27 |       <-- correct

The same happens for files written by DataFusion's own COPY. [NaN, 2.0, 3.0] (min ≠ max) and [NaN, NaN] (no min/max
written) are read correctly.

Both column orders are affected. The pyarrow files above have TYPE_DEFINED_ORDER and no nan_count. DataFusion/arrow-rs
60 writes IEEE_754_TOTAL_ORDER with nan_count, but still leaves NaN out of min/max: COPY (VALUES ('NaN'::double), (5.0))
gives min = max = 5.0, null_count = 0, nan_count = 1, and that file is read back as 5.0, 5.0. So DataFusion ignores a
nan_count that the file does carry.

Expected behavior

SELECT * FROM 'nan.parquet' returns NaN, -8.6e-27, as pyarrow, DuckDB and Polars do. Float statistics with min == max
should not be used to replace the column by a constant, because NaN values are not reflected in them.

Additional context

DataFusion main 290ad15 (2026-10-18), datafusion-cli 55.1.0 built from source; also 91cd7ff (2026-10-15). pyarrow 26.0.0.
Shapes like "one default value plus missing values as NaN" are common in sensor and ML feature data.
Related: Pruning of floating point Parquet columns is incorrect when NaN is present #15812 and draft PR fix: prevent incorrect Parquet pruning with NaNs #24804.

Willingness to contribute

None

Bound Parquet delete memory and match NaN rows

take

test(querier): pin that a nan in a promoted float column keeps its va…

@jensholdgaard

Written by Claude, an AI assistant, on behalf of @jensholdgaard. I ran the tests below and checked the results.

Another downstream project is affected: Ourios, a log
store on Parquet + DataFusion. It stores OpenTelemetry double attributes in Float64 columns,
and those values include NaN. With execution.collect_statistics = true, a stored NaN reads back
as the statistics constant, which changes what we return to users. We keep statistics
collection off until a release fixes this.

I ran the reproducer below against 56.0.0-rc1 and the current heads of both fix PRs. It writes
one row group holding 1.5, NaN, 1.5 with COPY … STORED AS PARQUET:

query (`collect_statistics = true`)	56.0.0-rc1	#27283 (`504998d`)	#27290 (`e2d5990`)	expected
`count(*) WHERE x < 2.0`	3	2	2	2
`count(*) WHERE isnan(x)`	0	1	1	1
`sum(x)`	4.5	NaN	NaN	NaN
`SELECT x`	1.5, 1.5, 1.5	1.5, NaN, 1.5	1.5, NaN, 1.5	1.5, NaN, 1.5

With collect_statistics = false, all three versions return the expected values. Both PRs
fix our case.

Reproducer

use datafusion::arrow::util::pretty::pretty_format_batches;
use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let dir = std::env::temp_dir().join("df-nan-repro");
    let _ = std::fs::remove_dir_all(&dir);
    std::fs::create_dir_all(&dir)?;
    // One row group: 1.5, NaN, 1.5. Parquet min/max leave NaN out, so min == max == 1.5.
    let file = dir.join("t.parquet");
    SessionContext::new()
        .sql(&format!(
            "COPY (SELECT column1 AS x FROM (VALUES (1.5::DOUBLE), ('NaN'::DOUBLE), (1.5::DOUBLE))) \
             TO '{}' STORED AS PARQUET",
            file.display()
        ))
        .await?
        .collect()
        .await?;
    let queries = [
        "SELECT count(*) AS below_2 FROM t WHERE x < 2.0",
        "SELECT count(*) AS nans FROM t WHERE isnan(x)",
        "SELECT sum(x) AS total FROM t",
        "SELECT x FROM t",
    ];
    for collect_statistics in [false, true] {
        let mut config = SessionConfig::new().with_target_partitions(1);
        config.options_mut().execution.collect_statistics = collect_statistics;
        let ctx = SessionContext::new_with_config(config);
        ctx.register_parquet("t", dir.to_str().unwrap(), ParquetReadOptions::default()).await?;
        println!("== collect_statistics = {collect_statistics}");
        for q in queries {
            let out = pretty_format_batches(&ctx.sql(q).await?.collect().await?)?.to_string();
            let values: Vec<_> = out.lines().skip(3).filter(|l| l.starts_with('|')).map(|l| l.trim_matches(|c| c == '|' || c == ' ')).collect();
            println!("  {q:<52} -> {values:?}");
        }
    }
    Ok(())
}

datafusion = { …, default-features = false, features = ["parquet", "sql", "nested_expressions"] }

Navigation Menu

Repository navigation

Plain `SELECT` of a Parquet float column returns wrong values: NaN rows are replaced by the statistics constant when min == max #26490

Describe the bug

To Reproduce

Expected behavior

Additional context

Willingness to contribute

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Plain SELECT of a Parquet float column returns wrong values: NaN rows are replaced by the statistics constant when min == max #26490

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Willingness to contribute

Activity

compiler-segfault commented on Nov 10, 2026

jensholdgaard commented on Nov 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Plain `SELECT` of a Parquet float column returns wrong values: NaN rows are replaced by the statistics constant when min == max #26490