Skip to content

Commit f0ac4d3

Browse files
docs(rfc): keep datafusion statistics collection off by design (#1279)
* docs(rfc): keep datafusion statistics collection off by design RFC 0021 §3.2a turned execution.collect_statistics off as a workaround for apache/datafusion#24769, to drop once the fix shipped. Testing 56.0.0-rc1 showed two open upstream bugs in the same statistics-to-constant path: NaN rows of a float column read back as the min == max statistic (#26490, wrong values for promoted Float64 attributes), and pruning still lost when the predicate holds a term statistics can't evaluate (#27415, RFC0044.7). The new §3.2c makes statistics off the intended design, says what it costs (plan-time file statistics only; row-group pruning still runs), and requires a later amendment with both bugs fixed, the guard tests green and a B1/B2 run before turning it on. RFC0021.10 pins the NaN case. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0125HXBoGeZ4wPFs9hDidYSx Signed-off-by: Jens Holdgaard Pedersen <[email protected]> * docs(rfc): separate the nan behaviour test from the stats-off guard RFC0021.10 named the statistics-off setting assertion alongside the NaN behaviour test, so the criterion could never pass with statistics on. The re-enable conditions now require only the behaviour test green, and the later amendment rewrites the off-only clause and its assertion. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0125HXBoGeZ4wPFs9hDidYSx Signed-off-by: Jens Holdgaard Pedersen <[email protected]> * test(querier): cite rfc0021.10 in the nan guard test Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0125HXBoGeZ4wPFs9hDidYSx Signed-off-by: Jens Holdgaard Pedersen <[email protected]> --------- Signed-off-by: Jens Holdgaard Pedersen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 54c2a59 commit f0ac4d3

2 files changed

Lines changed: 74 additions & 2 deletions

File tree

‎crates/ourios-querier/tests/it/nan_promoted_float.rs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@
66
//! constant and returns the statistic for the NaN rows
77
//! (apache/datafusion#26490). The querier keeps statistics collection off;
88
//! this test pins the behaviour itself, so a `DataFusion` upgrade or a
9-
//! session change that brings the substitution back fails here.
9+
//! session change that brings the substitution back fails here. This is
10+
//! the behaviour test of RFC0021.10.
1011
1112
use ourios_parquet::{PromotedAttributes, PromotedClass, PromotedKey};
1213

‎docs/rfcs/0021-datafusion-arrow-upgrade.md‎

Lines changed: 72 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,61 @@ the original phase 2 became available on its own:
186186
the RFC 0009 manifest plus partition-directory windowing rather than
187187
from DataFusion's collected file statistics. Revisit when the
188188
upstream fix ships in a release we depend on — the same three
189-
pruning tests are the gate in both directions.
189+
pruning tests are the gate in both directions. *(2026-11-14:
190+
superseded by §3.2c. Statistics off is now the intended design,
191+
and turning them on needs its own amendment.)*
192+
193+
### 3.2c Statistics collection stays off by design (amendment 2026-11-14)
194+
195+
*Maintainer-approved amendment.* §3.2a treated
196+
`execution.collect_statistics = false` as a workaround to drop once
197+
apache/datafusion#24769 was fixed. Testing 56.0.0-rc1 showed it is
198+
protecting more than pruning, so it is now a design decision.
199+
200+
**Why it stays off.** With statistics on, DataFusion replaces a column
201+
whose file statistics show `min == max` and no nulls with that
202+
constant. Two upstream bugs sit in that path:
203+
204+
- **Wrong values** (apache/datafusion#26490, open). Parquet float
205+
min/max leave NaN out, so a float column of one value plus NaNs looks
206+
constant, and the NaN rows read back as that value. Ourios writes
207+
OTLP `DoubleValue`s, NaN included, into promoted `Float64` attribute
208+
columns (RFC 0042). With statistics on, those rows would read back
209+
wrong, which breaks OTLP fidelity. Other projects built on DataFusion
210+
have hit this too (nautilus_trader, ltseq, delta-rs).
211+
- **Lost pruning** (apache/datafusion#27415, open). #24770, in 56.0.0,
212+
restores pruning when the folded predicate is a bare constant, but not
213+
when it still holds a term statistics can't evaluate. RFC 0044's
214+
body-equality rewrite has such terms
215+
(`array_element(separators, n) = '…'`), so RFC0044.7 still fails with
216+
statistics on.
217+
218+
**What off costs, and what it doesn't.** Row-group pruning from each
219+
file's footer, the pruning pillar #1 relies on, runs either way. Off
220+
gives up only plan-time file statistics:
221+
- constant-column substitution;
222+
- answering a count from metadata;
223+
- skipping a whole file before opening it.
224+
225+
It also saves a footer read per file at plan time, which a many-file
226+
log store would otherwise pay on every query. Ourios gets file-level
227+
selection from the RFC 0009 manifest and partition windowing instead.
228+
229+
**Turning statistics on** takes a later amendment, not an upstream
230+
release. It must show:
231+
1. #26490 and #27415 fixed in a DataFusion release we depend on;
232+
2. RFC0021.10's NaN behaviour test and the RFC0044.7/.8 and RFC0007.1
233+
pruning tests green with statistics on (RFC0007.1's "reads bytes"
234+
guard reworded, since a count over a statistics-constant column
235+
legitimately reads none);
236+
3. an indicative B1/B2 run on a many-file corpus where the plan-time
237+
footer reads pay for themselves.
238+
239+
That amendment also rewrites RFC0021.10's statistics-off clause and the
240+
`the_shared_session_disables_statistics_collection` assertion it names,
241+
which pin the setting itself and would otherwise fail by construction.
242+
243+
Scenario RFC0021.10 is the criterion. The status stays `green`.
190244

191245
### 3.2b Phase 2b — object_store ≥ 0.14 (still upstream-gated)
192246

@@ -298,6 +352,20 @@ back to `red`) the day a released DataFusion accepts object_store
298352
> Then no `thrift` crate is present (#295 closed).
299353
> Asserted by `rfc0021_8_thrift_is_absent_from_the_lockfile`.
300354
355+
> **Scenario RFC0021.10 — a statistics constant never replaces data**
356+
> (amendment 2026-11-14, §3.2c).
357+
> Given a promoted `Float64` attribute column holding `[1.5, NaN, 1.5]`
358+
> in one file,
359+
> When the querier's production session filters `attr.cost_usd < 2.0`
360+
> and sums the column,
361+
> Then the filter counts 2 rows and the sum is reported as NULL
362+
> (non-finite), never 3 rows or a finite 4.5. Read as the `min == max`
363+
> statistic, the NaN row would count as 1.5. Behaviour test:
364+
> `nan_promoted_float::a_nan_in_a_promoted_float_column_is_never_read_as_the_statistics_constant`.
365+
> Until a later amendment turns statistics on (§3.2c), the session also
366+
> keeps `collect_statistics` off, asserted by
367+
> `exec::session_tests::the_shared_session_disables_statistics_collection`.
368+
301369
> **Scenario RFC0021.9 (phase 2a half) — supply chain re-cleared.**
302370
> Given the new transitive set,
303371
> When `cargo deny check` runs,
@@ -357,6 +425,9 @@ one artefact and changes no test semantics:
357425

358426
## 8. References
359427

428+
- apache/datafusion#24769 and #24770 (the §3.2a regression and its
429+
partial fix), #27415 (the remaining pruning case), #26490 (NaN rows
430+
read as the statistics constant) — §3.2c.
360431
- Epic #314 (this RFC), #310 (object_store 0.14, blocked), #295
361432
(thrift GHSA-2f9f-gq7v-9h6m), #276 (RFC 0017 dual decoder), #313
362433
(renovate hold).

0 commit comments

Comments
 (0)