You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 647d96d
Browse filesBrowse the repository at this point in the historyBrowse files
DataFusion 55.0.0 fired RFC 0021's own reopening trigger, but only
half of it: DF 55 carries arrow/parquet 59.2 while still pinning
object_store ^0.13.2. Phase 2 is therefore split (§3.2a/§3.2b) and
this lands 2a.
- DataFusion 54 -> 55, arrow/parquet 58 -> 59, MSRV 1.88 -> 1.94.
- `thrift` leaves the lockfile (parquet 59 dropped it), clearing
GHSA-2f9f-gq7v-9h6m (#295) and the two Dependabot alerts previously
dismissed as risk-tolerated. `paste` leaves too, so its dead
RUSTSEC-2024-0436 ignore is removed from deny.toml.
- API churn was two items: an arrow-cast pin, and DF 55's new required
ExecutionPlan::apply_expressions (three test doubles in
ourios-df-otel). No production code changed for the bump itself.
- MSRV 1.94 enables clippy's duration_suboptimal_units, which suggests
Duration::from_days/from_mins — both unstable (rust#120301) — so it
is allowed workspace-wide with that reason rather than chased into
nightly-only APIs.
RFC0021.1 is re-baselined to assert the invariant (exactly one arrow
major, exactly one datafusion) plus floors (arrow >= 59, DF >= 55)
rather than version literals: a literal-pinning test is a
change-detector that fails on every intentional upgrade and catches
nothing the invariant misses. RFC0021.8 lands as a real test
asserting thrift's absence from the lockfile.
One behavioural decision, recorded in §3.2a: the querier runs with
execution.collect_statistics off, via a single exec::session() shared
by both production session sites. DF 55 substitutes columns that
per-file statistics prove constant, and the all-NULL case folds to a
NULL literal, collapsing `body == "…"` to a bare constant — leaving
no column reference for a pruning predicate, so RFC 0044's
body-equality queries scanned every row group instead of skipping
them (pillar #1; caught by RFC0044.7/.8 and RFC0007.1). Reported as
apache/datafusion#24769 with a fix proposed in #24770, bisected to
DataFusion #22969. The override also avoids a per-file footer read at
plan time, which a many-file log store pays on every query; Ourios
derives its pruning from the RFC 0009 manifest and partition-directory
windowing, not from DataFusion's collected file statistics. The same
three tests gate the decision in both directions.
Phase 2b (object_store >= 0.14, RFC0021.7 and the rest of .9) stays
upstream-gated; epic #314 tracks it.
Verified: workspace 1448/1448, clippy pedantic zero, fmt, mdbook,
cargo deny clean.
BREAKING CHANGE: the workspace MSRV moves from 1.88 to 1.94, and the
storage/query stack moves to DataFusion 55 / arrow 59 / parquet 59.
Signed-off-by: Jens Holdgaard Pedersen <[email protected]>
0 commit comments