Skip to content

Commit b764605

Browse files
committed
fix(querier): run queries with collect_statistics off
Unblocks RFC 0021 phase 2a. DataFusion 55 substitutes columns that per-file statistics prove constant; the all-NULL case folds to a NULL literal, which collapses `body == "…"` to a bare constant and leaves no column reference for a pruning predicate — so RFC 0044's body-equality queries scanned every row group instead of skipping them. Pillar #1, caught by RFC0044.7/.8 and RFC0007.1. Upstream: apache/datafusion#24769, fix proposed in #24770, bisected to DataFusion #22969 (which made collect_statistics the session default). Turning the option off restores the pruning exactly. Both production session sites now go through one `exec::session()` so the setting cannot drift between the query and drift paths. The override is not purely defensive: collection also costs a footer read per file at plan time — a cost a many-file log store pays on every query — and Ourios derives its pruning from the RFC 0009 manifest plus partition-directory windowing, not from DataFusion's collected file statistics. RFC 0021 §3.2a records the decision and names the revisit trigger; the same three pruning tests gate it in both directions. Verified: workspace 1448/1448, clippy pedantic zero, fmt, mdbook. Signed-off-by: Jens Holdgaard Pedersen <[email protected]>
1 parent 0ec0a87 commit b764605

4 files changed

Lines changed: 46 additions & 4 deletions

File tree

‎crates/ourios-querier/src/drift.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,7 @@ async fn audit_dataframe(
212212
files: &audit_scan::AuditFiles,
213213
tenant: &TenantId,
214214
) -> Result<(SessionContext, datafusion::prelude::DataFrame), QueryError> {
215-
let ctx = SessionContext::new();
215+
let ctx = crate::exec::session();
216216
let urls = audit_table_urls(&ctx, backend, files)?;
217217
// Bare inference is an explicit choice here: audit files share one
218218
// schema by construction — see `SchemaMode::Infer` (epic #745

‎crates/ourios-querier/src/exec.rs‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,28 @@ pub(crate) enum SchemaMode<'a> {
3636
Infer,
3737
}
3838

39+
/// The session every query path runs on.
40+
///
41+
/// `execution.collect_statistics` is **off** deliberately (RFC 0021
42+
/// §3.2a). With it on, `DataFusion` substitutes columns that per-file
43+
/// statistics prove constant — and folding an all-NULL column to a
44+
/// NULL literal collapses predicates like `body == "…"` to a bare
45+
/// constant, which leaves no column reference for a pruning predicate
46+
/// to be built over, so the row groups are scanned instead of skipped
47+
/// (upstream apache/datafusion#24769, fix proposed in #24770). That
48+
/// defeats pillar #1 on exactly the queries RFC 0044 exists to make
49+
/// fast. Collection also costs a per-file footer read at plan time,
50+
/// which a many-file log store pays on every query.
51+
///
52+
/// Remove this override once the upstream fix ships in a release we
53+
/// depend on; the RFC0044.7/.8 and RFC0007.1 pruning tests are the
54+
/// gate that will catch it either way.
55+
pub(crate) fn session() -> SessionContext {
56+
let mut config = SessionConfig::new();
57+
config.options_mut().execution.collect_statistics = false;
58+
SessionContext::new_with_config(config)
59+
}
60+
3961
/// Register `urls` as listing table `name` on `ctx` and return its
4062
/// `DataFrame`. **Tenancy** is not a parameter here by design: the
4163
/// §3.7 scope is enforced where URLs are *resolved* (the tenant-

‎crates/ourios-querier/src/lib.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ use datafusion::datasource::listing::{
7373
use datafusion::error::DataFusionError;
7474
use datafusion::functions_aggregate::expr_fn::count;
7575
use datafusion::physical_plan::metrics::{MetricValue, MetricsSet};
76-
use datafusion::prelude::{SessionContext, col, lit};
76+
use datafusion::prelude::{SessionConfig, SessionContext, col, lit};
7777
use ourios_core::tenant::TenantId;
7878
use ourios_parquet::columns;
7979
use ourios_parquet::hour_partition_in_window;
@@ -655,7 +655,7 @@ impl Querier {
655655
let enc = percent_encode_tenant(tenant.as_str());
656656
let data_prefix = format!("data/tenant_id={enc}");
657657

658-
let ctx = SessionContext::new();
658+
let ctx = crate::exec::session();
659659
// Resolve the live file set under the tenant's `data/` prefix,
660660
// honouring the RFC 0009 §3.4 manifest (glob-fallback when absent),
661661
// and produce the per-file table URLs (local absolute path, or

‎docs/rfcs/0021-datafusion-arrow-upgrade.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ superseded-by: —
1111

1212
# RFC 0021 — Coordinated DataFusion / Arrow upgrade, phased behind upstream
1313

14-
> **Status: `green` (2026-08-28) — phase 2a landed; phase 2b still
14+
> **Status: `green` (2026-08-29) — phase 2a complete; phase 2b still
1515
> upstream-gated.** DataFusion 55.0.0 shipped and fired this RFC's own
1616
> reopening trigger, but **only half of it**: DF 55 carries
1717
> arrow/parquet **59.2** yet still pins `object_store ^0.13.2`. Phase 2
@@ -167,6 +167,26 @@ the original phase 2 became available on its own:
167167
- API churn was two items: an `arrow-cast` pin, and DataFusion 55's new
168168
required `ExecutionPlan::apply_expressions` (three test doubles in
169169
`ourios-df-otel`). No production code changed.
170+
- **`execution.collect_statistics` is turned off on the querier's
171+
session** (`ourios_querier::exec::session`), which is the one
172+
behavioural decision in this phase. DataFusion 55 substitutes
173+
columns that per-file statistics prove constant; the all-NULL case
174+
folds to a NULL literal, collapsing predicates like
175+
`body == "…"` to a bare constant, which leaves no column reference
176+
for a pruning predicate — so RFC 0044's body-equality queries scan
177+
every row group instead of skipping them (pillar #1, and the
178+
RFC0044.7/.8 + RFC0007.1 tests caught it). Upstream bug
179+
[apache/datafusion#24769]; fix proposed in
180+
[apache/datafusion#24770], bisected to DataFusion #22969 which
181+
made `collect_statistics` the session-level default.
182+
183+
The override is not purely defensive: statistics collection also
184+
costs a footer read **per file at plan time**, which a many-file log
185+
store pays on every query, and Ourios derives its own pruning from
186+
the RFC 0009 manifest plus partition-directory windowing rather than
187+
from DataFusion's collected file statistics. Revisit when the
188+
upstream fix ships in a release we depend on — the same three
189+
pruning tests are the gate in both directions.
170190

171191
### 3.2b Phase 2b — object_store ≥ 0.14 (still upstream-gated)
172192

0 commit comments

Comments
 (0)