Skip to content

Commit 647d96d

Browse files
committed
refactor(deps)!: datafusion 55 / arrow 59 / parquet 59 (RFC 0021 phase 2a)
DataFusion 55.0.0 fired RFC 0021's own reopening trigger, but only half of it: DF 55 carries arrow/parquet 59.2 while still pinning object_store ^0.13.2. Phase 2 is therefore split (§3.2a/§3.2b) and this lands 2a. - DataFusion 54 -> 55, arrow/parquet 58 -> 59, MSRV 1.88 -> 1.94. - `thrift` leaves the lockfile (parquet 59 dropped it), clearing GHSA-2f9f-gq7v-9h6m (#295) and the two Dependabot alerts previously dismissed as risk-tolerated. `paste` leaves too, so its dead RUSTSEC-2024-0436 ignore is removed from deny.toml. - API churn was two items: an arrow-cast pin, and DF 55's new required ExecutionPlan::apply_expressions (three test doubles in ourios-df-otel). No production code changed for the bump itself. - MSRV 1.94 enables clippy's duration_suboptimal_units, which suggests Duration::from_days/from_mins — both unstable (rust#120301) — so it is allowed workspace-wide with that reason rather than chased into nightly-only APIs. RFC0021.1 is re-baselined to assert the invariant (exactly one arrow major, exactly one datafusion) plus floors (arrow >= 59, DF >= 55) rather than version literals: a literal-pinning test is a change-detector that fails on every intentional upgrade and catches nothing the invariant misses. RFC0021.8 lands as a real test asserting thrift's absence from the lockfile. One behavioural decision, recorded in §3.2a: the querier runs with execution.collect_statistics off, via a single exec::session() shared by both production session sites. DF 55 substitutes columns that per-file statistics prove constant, and the all-NULL case folds to a NULL literal, collapsing `body == "…"` to a bare constant — leaving no column reference for a pruning predicate, so RFC 0044's body-equality queries scanned every row group instead of skipping them (pillar #1; caught by RFC0044.7/.8 and RFC0007.1). Reported as apache/datafusion#24769 with a fix proposed in #24770, bisected to DataFusion #22969. The override also avoids a per-file footer read at plan time, which a many-file log store pays on every query; Ourios derives its pruning from the RFC 0009 manifest and partition-directory windowing, not from DataFusion's collected file statistics. The same three tests gate the decision in both directions. Phase 2b (object_store >= 0.14, RFC0021.7 and the rest of .9) stays upstream-gated; epic #314 tracks it. Verified: workspace 1448/1448, clippy pedantic zero, fmt, mdbook, cargo deny clean. BREAKING CHANGE: the workspace MSRV moves from 1.88 to 1.94, and the storage/query stack moves to DataFusion 55 / arrow 59 / parquet 59. Signed-off-by: Jens Holdgaard Pedersen <[email protected]>
1 parent fe2b644 commit 647d96d

17 files changed

Lines changed: 393 additions & 240 deletions

File tree

‎Cargo.lock‎

Lines changed: 156 additions & 165 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ exclude = ["fuzz"]
2828
# bumps this one line; the matching `vX.Y.Z` tag then drives cargo-dist + image.
2929
version = "0.9.0"
3030
edition = "2024"
31-
rust-version = "1.88"
31+
rust-version = "1.94"
3232
license = "Apache-2.0"
3333
repository = "https://github.com/jensholdgaard/ourios"
3434

@@ -40,6 +40,12 @@ unsafe_code = "deny"
4040
# All member crates declare `[lints] workspace = true`, so this
4141
# propagates without per-crate edits; `cargo doc --workspace
4242
# --no-deps --all-features` is the local check.
43+
# `duration_suboptimal_units` (new in the toolchain the 1.94 MSRV admits)
44+
# suggests `Duration::from_days` / `from_mins`, which are **unstable**
45+
# (rust-lang/rust#120301) — the lint is un-actionable on stable, and the
46+
# flagged sites are honest anyway: `from_secs(86_400)` day-scale sentinels
47+
# and Unix-epoch fixture stamps, both of which read better in seconds.
48+
# Revisit when the constructors stabilise.
4349
[workspace.lints.rustdoc]
4450
broken_intra_doc_links = "deny"
4551
private_intra_doc_links = "deny"
@@ -48,6 +54,12 @@ bare_urls = "warn"
4854

4955
[workspace.lints.clippy]
5056
pedantic = { level = "warn", priority = -1 }
57+
# `duration_suboptimal_units` suggests `Duration::from_days` / `from_mins`,
58+
# which are **unstable** (rust-lang/rust#120301) — un-actionable on stable.
59+
# The flagged sites are honest in seconds anyway: `from_secs(86_400)`
60+
# day-scale "effectively never" sentinels and Unix-epoch fixture stamps.
61+
# Revisit when the constructors stabilise.
62+
duration_suboptimal_units = "allow"
5163

5264
# The profile that 'dist' will build with
5365
[profile.dist]

‎crates/ourios-bench/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ tokio = { version = "=1.53.1", default-features = false, features = ["rt-multi-t
111111
# tests: opens a comparative-store file's Parquet footer directly to
112112
# check the promoted column the writer declared. Same major as the
113113
# workspace arrow/parquet tree, so Cargo.lock pins one copy.
114-
parquet = { version = "=58.4.0", default-features = false, features = ["arrow"] }
114+
parquet = { version = "=59.2.0", default-features = false, features = ["arrow"] }
115115
# RFC0024.2 draws calibrated records through a fixed-seed
116116
# `TestRunner` (the strategies come from `ourios-testgen`, already a
117117
# regular dependency above).

‎crates/ourios-df-otel/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ path = "src/lib.rs"
1616
# Pinned to the same DataFusion 54 the rest of the workspace runs (RFC 0021).
1717
# No `ourios-*` dep — deliberate, so this lifts cleanly to a standalone
1818
# `datafusion-contrib` crate (RFC 0040 §3.5).
19-
datafusion = { version = "54", default-features = false, features = ["parquet"] }
19+
datafusion = { version = "55", default-features = false, features = ["parquet"] }
2020
opentelemetry = { version = "0.32", default-features = false, features = ["trace"] }
2121

2222
[dev-dependencies]

‎crates/ourios-df-otel/benches/unsampled_walk.rs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,19 @@ impl ExecutionPlan for PanicsIfTouched {
6262
panic!("record_plan_spans must not walk the plan on the unsampled path")
6363
}
6464

65+
// DataFusion 55 made this a required method; like `children`, the
66+
// unsampled path must never reach it.
67+
fn apply_expressions(
68+
&self,
69+
_f: &mut dyn FnMut(
70+
&Arc<dyn datafusion::physical_expr::PhysicalExpr>,
71+
) -> datafusion::error::Result<
72+
datafusion::common::tree_node::TreeNodeRecursion,
73+
>,
74+
) -> datafusion::error::Result<datafusion::common::tree_node::TreeNodeRecursion> {
75+
panic!("record_plan_spans must not walk the plan on the unsampled path")
76+
}
77+
6578
fn with_new_children(
6679
self: Arc<Self>,
6780
_children: Vec<Arc<dyn ExecutionPlan>>,

‎crates/ourios-df-otel/src/lib.rs‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,19 @@ mod tests {
308308
vec![]
309309
}
310310

311+
// DataFusion 55 made this a required method: a leaf carries no
312+
// physical expressions, so the walk continues untouched.
313+
fn apply_expressions(
314+
&self,
315+
_f: &mut dyn FnMut(
316+
&Arc<dyn datafusion::physical_expr::PhysicalExpr>,
317+
) -> datafusion::error::Result<
318+
datafusion::common::tree_node::TreeNodeRecursion,
319+
>,
320+
) -> datafusion::error::Result<datafusion::common::tree_node::TreeNodeRecursion> {
321+
Ok(datafusion::common::tree_node::TreeNodeRecursion::Continue)
322+
}
323+
311324
fn with_new_children(
312325
self: Arc<Self>,
313326
_children: Vec<Arc<dyn ExecutionPlan>>,
@@ -439,6 +452,19 @@ mod tests {
439452
panic!("record_plan_spans must not walk the plan on the unsampled path")
440453
}
441454

455+
// DataFusion 55 made this a required method: a leaf carries no
456+
// physical expressions, so the walk continues untouched.
457+
fn apply_expressions(
458+
&self,
459+
_f: &mut dyn FnMut(
460+
&Arc<dyn datafusion::physical_expr::PhysicalExpr>,
461+
) -> datafusion::error::Result<
462+
datafusion::common::tree_node::TreeNodeRecursion,
463+
>,
464+
) -> datafusion::error::Result<datafusion::common::tree_node::TreeNodeRecursion> {
465+
Ok(datafusion::common::tree_node::TreeNodeRecursion::Continue)
466+
}
467+
442468
fn with_new_children(
443469
self: Arc<Self>,
444470
_children: Vec<Arc<dyn ExecutionPlan>>,

‎crates/ourios-ingester/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ tempfile = "=3.27.0"
145145
# Read flushed files' Parquet schemas back in the RFC 0022 promoted-set
146146
# threading test (pinned in lockstep with ourios-parquet's arrow stack,
147147
# matching ourios-querier's dev pin).
148-
parquet = { version = "=58.4.0", default-features = false, features = ["arrow"] }
148+
parquet = { version = "=59.2.0", default-features = false, features = ["arrow"] }
149149
# Drive the axum `Router` in-process (`ServiceExt::oneshot`) for the
150150
# HTTP-transport acceptance tests — no real socket.
151151
tower = { version = "=0.5.3", default-features = false, features = ["util"] }

‎crates/ourios-parquet/Cargo.toml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,9 @@ ourios-core = { path = "../ourios-core" }
1818
# `parquet::arrow::ArrowWriter` / `ParquetRecordBatchReader`).
1919
# `parquet` carries the actual writer (`arrow` feature pulls in
2020
# `arrow::array::RecordBatch` integration).
21-
arrow-array = { version = "58", default-features = false }
22-
arrow-schema = { version = "58", default-features = false }
23-
parquet = { version = "58", default-features = false, features = ["arrow", "zstd"] }
21+
arrow-array = { version = "59", default-features = false }
22+
arrow-schema = { version = "59", default-features = false }
23+
parquet = { version = "59", default-features = false, features = ["arrow", "zstd"] }
2424
# Object-storage backend (RFC 0013) — one trait over LocalFileSystem (dev/
2525
# test) and S3-compatible stores. Already in the workspace tree via
2626
# DataFusion (the querier); declared here directly as the storage seam's
@@ -84,7 +84,7 @@ ourios-testgen = { path = "../ourios-testgen" }
8484
ourios-miner = { path = "../ourios-miner" }
8585
# RFC0021.4 view-representation test: cast a plain batch's string/binary
8686
# columns to Utf8View/BinaryView and prove the shared decoder reads both.
87-
arrow-cast = { version = "=58.4.0", default-features = false }
87+
arrow-cast = { version = "=59.2.0", default-features = false }
8888
# Property tests for the compaction row-conservation invariant
8989
# (RFC0009.2 / CLAUDE.md §6.2 — invariants get a property test).
9090
proptest = "=1.11.0"

‎crates/ourios-parquet/tests/it/rfc0021_arrow_upgrade.rs‎

Lines changed: 52 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -177,12 +177,21 @@ fn rfc0021_2_pre_upgrade_fixture_reads_identically() {
177177
/// Scenario RFC0021.1 — one arrow.
178178
/// See `docs/rfcs/0021-datafusion-arrow-upgrade.md` §5.
179179
///
180-
/// Inspects the committed lockfile: every `arrow*` package is on the
181-
/// single post-upgrade major, `datafusion` is on 54.x, and the workspace
182-
/// MSRV is 1.88. A second arrow major sneaking back in via a transitive
183-
/// bump fails here, not in a human review of a renovate diff.
180+
/// Inspects the committed lockfile for the *invariant* the RFC actually
181+
/// bought: every `arrow*` package sits on **one** major and there is
182+
/// exactly **one** `datafusion`. A second arrow major sneaking back in
183+
/// via a transitive bump fails here, not in a human review of a
184+
/// renovate diff — that dual-major state is what forced RFC 0017's
185+
/// dual decoder and the `schema_force_view_types` override (#276).
186+
///
187+
/// It deliberately does **not** pin the major to a literal. A test that
188+
/// asserts "we are on arrow 58" is a change-detector: it fails on every
189+
/// intentional upgrade (as it did on the phase-2a bump) and catches no
190+
/// bug the invariant misses. The literal versions live where they are
191+
/// executable — `Cargo.toml` and the lockfile — and the *floor* the
192+
/// security posture depends on is asserted separately below.
184193
#[test]
185-
fn rfc0021_1_one_arrow_major_and_datafusion_54() {
194+
fn rfc0021_1_one_arrow_major_and_one_datafusion() {
186195
let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
187196
.parent()
188197
.and_then(std::path::Path::parent)
@@ -209,26 +218,55 @@ fn rfc0021_1_one_arrow_major_and_datafusion_54() {
209218
datafusion_versions.push(version.to_string());
210219
}
211220
}
221+
let majors: Vec<_> = arrow_majors.into_iter().collect();
212222
assert_eq!(
213-
arrow_majors.into_iter().collect::<Vec<_>>(),
214-
["58"],
215-
"exactly one arrow major (58.x) in the lockfile"
223+
majors.len(),
224+
1,
225+
"exactly one arrow major in the lockfile, got {majors:?}"
216226
);
217227
assert_eq!(
218228
datafusion_versions.len(),
219229
1,
220230
"exactly one datafusion in the lockfile, got {datafusion_versions:?}"
221231
);
232+
233+
// Floors, not pins: phase 2a's security clearance depends on being
234+
// at or above these, and nothing depends on being *at* them. A
235+
// routine upgrade passes; a downgrade that would drag `thrift` back
236+
// in fails.
237+
let major: u32 = majors[0].parse().expect("arrow major is numeric");
222238
assert!(
223-
datafusion_versions[0].starts_with("54."),
224-
"datafusion 54.x, got {}",
225-
datafusion_versions[0]
239+
major >= 59,
240+
"arrow major >= 59 (phase 2a floor), got {major}"
226241
);
242+
let df_major: u32 = datafusion_versions[0]
243+
.split('.')
244+
.next()
245+
.and_then(|m| m.parse().ok())
246+
.expect("datafusion major is numeric");
247+
assert!(
248+
df_major >= 55,
249+
"datafusion major >= 55 (phase 2a floor), got {df_major}"
250+
);
251+
}
227252

228-
let manifest = std::fs::read_to_string(root.join("Cargo.toml")).expect("read Cargo.toml");
253+
/// Scenario RFC0021.8 (phase 2a) — thrift is gone.
254+
/// See `docs/rfcs/0021-datafusion-arrow-upgrade.md` §5.
255+
///
256+
/// parquet 59 dropped its `thrift` dependency, which is the whole
257+
/// security case for phase 2a (GHSA-2f9f-gq7v-9h6m, #295). Asserted on
258+
/// the lockfile so a transitive re-introduction fails here rather than
259+
/// in a Dependabot alert weeks later.
260+
#[test]
261+
fn rfc0021_8_thrift_is_absent_from_the_lockfile() {
262+
let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
263+
.parent()
264+
.and_then(std::path::Path::parent)
265+
.expect("workspace root is two levels above CARGO_MANIFEST_DIR");
266+
let lock = std::fs::read_to_string(root.join("Cargo.lock")).expect("read Cargo.lock");
229267
assert!(
230-
manifest.contains("rust-version = \"1.88\""),
231-
"workspace MSRV is 1.88"
268+
!lock.contains("name = \"thrift\""),
269+
"thrift must stay out of the lockfile (GHSA-2f9f-gq7v-9h6m, #295)"
232270
);
233271
}
234272

‎crates/ourios-querier/Cargo.toml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ percent-encoding = { version = "2", default-features = false, features = ["alloc
4848
# `nested_expressions` registers the list functions the §6.3 `param(n)`
4949
# group term lowers through (`array_element` over the RFC 0005 `params`
5050
# list column).
51-
datafusion = { version = "54", default-features = false, features = [
51+
datafusion = { version = "55", default-features = false, features = [
5252
"parquet",
5353
"string_expressions",
5454
"regex_expressions",
@@ -132,9 +132,9 @@ testcontainers-modules = { version = "=0.15.0", features = ["localstack"] }
132132
# over a batch built from the public `data_schema()`. Pinned to
133133
# the same arrow/parquet 58 the whole workspace now shares
134134
# (RFC 0021 phase 1 — the pre-upgrade cross-version handoff is gone).
135-
parquet = { version = "=58.4.0", default-features = false, features = ["arrow"] }
136-
arrow-array = { version = "=58.4.0", default-features = false }
137-
arrow-schema = { version = "=58.4.0", default-features = false }
135+
parquet = { version = "=59.2.0", default-features = false, features = ["arrow"] }
136+
arrow-array = { version = "=59.2.0", default-features = false }
137+
arrow-schema = { version = "=59.2.0", default-features = false }
138138
# RFC0002.10 — proves the canonical β serialisation survives a YAML
139139
# scalar round-trip (the Perses-embedding guarantee). Test-only.
140140
serde_yaml_ng = { version = "=0.10.0", default-features = false }

0 commit comments

Comments
 (0)