chore(deps): prepare for DataFusion 56 (draft until release) by jensholdgaard · Pull Request #846 · jensholdgaard/ourios

jensholdgaard · 2026-09-27T14:25:50Z

Summary

This draft builds Ourios against the apache/datafusion 56.0.0-rc1 tag (cce6374, since 4858ce2), via [patch.crates-io]. That release candidate carries apache/datafusion#24770, the fix for the #24769 row-group pruning regression. The branch also moves the workspace to arrow/parquet 60.1.0 and object_store 0.14.2. It stays a draft until DataFusion 56.0.0 is on crates.io.

Commits:

37b8eac chore(deps): build against upstream datafusion main with arrow 60 (maintainer's commit). I re-checked its claim: cargo clippy --all-targets --all-features -D warnings is clean on it with no source changes.
2bf6b44 fix(serving): pin the client-cert verifier to the ring provider.
- object_store 0.14's aws feature hard-enables rustls/aws-lc-rs. With ring also on, rustls can no longer pick a provider automatically.
- As a result, WebPkiClientVerifier::builder() panicked on the mTLS path, failing RFC0030.4 and RFC0030.8.
- Merged to main as fix(serving): pin the client-cert verifier to the ring provider #845, so this commit drops out as a no-op.
18fdb31 chore(deps): admit the datafusion git source and the webpki-root-certs data licence.
- deny.toml gets allow-git for exactly https://github.com/apache/datafusion.
- webpki-root-certs is new, arriving via reqwest 0.13 and rustls-platform-verifier. It carries the same Mozilla CA data under the same CDLA-Permissive-2.0 licence as webpki-roots, so it gets the same per-crate exception in deny.toml and about.toml. (Main has since added the same exception itself.)
- THIRD-PARTY-LICENSES.md is regenerated with cargo-about 0.9.1.
65a588d chore(fuzz): mirror the patch into the standalone fuzz workspace.
- The fuzz workspace depends on ourios-querier but doesn't inherit the root patch. Its lockfile went stale and failed the --locked step in the cargo deny job.
- Unpatched, it would also mix DataFusion 55 (arrow 59) with arrow 60.
- After the change, cargo check --manifest-path fuzz/Cargo.toml --locked is clean.
Merges of main. These keep this branch's lockfile and versions (arrow/parquet 60, object_store 0.14) and take main's other bumps. Before rc1 they also repointed datafusion at the patched revision when main's lockfile locked a newer crates.io 55.x (cargo update -p [email protected] --precise 55.1.0).
- 1eb0789 also moves this branch's zstd 0.14 line (pulled in by arrow 60) to the patched 0.14.3 / zstd-safe 8.2.0 for RUSTSEC-2026-0391. Main can only waive that advisory, because it is still on zstd 0.13.
- fda45ab keeps arrow/parquet 60.0.0 over main's 59.3.2 bump.
Two merge commits carry a small source change each, because object_store 0.14 added a GetResult::extensions field:
- 8314897: main's MeteredStore::get_opts (feat: object-store client, blocking-bridge and async-runtime metrics #937, crates/ourios-parquet/src/store/metered.rs) now carries extensions through the wrapper, keeping the HTTP response extensions.
- 08ef265: main's ListedSizes HEAD answer (perf(querier): answer the per-file head from the resolving listing #1131, crates/ourios-querier/src/listed_sizes.rs) sets extensions to the default.
28ec00a (maintainer) test(parquet): split the decoder-panic fixtures parquet 60 now rejects.
4858ce2 chore(deps): build against the datafusion 56.0.0-rc1 tag.
- The patch moves from the main revision f029b944 to the rc1 commit cce6374, in both the root and fuzz/.
- The patched crates report 56.0.0, so ourios-querier and ourios-df-otel now ask for datafusion = "56".
- The arrow/parquet pins move from =60.0.0 to =60.1.0. object_store stays =0.14.2.

API breaks

Rust API (DataFusion 56, arrow 60.1, object_store 0.14): one small break.
- DataFusion 56 and arrow 60 needed no source changes.
- object_store 0.14 adds the public field GetResult::extensions. Code that destructures or builds a GetResult must name it. Two sites on main do: MeteredStore::get_opts (feat: object-store client, blocking-bridge and async-runtime metrics #937) and ListedSizes (perf(querier): answer the per-file head from the resolving listing #1131), both adapted in commit 6.
- These edits are 0.14-only: they ride this PR and can't be split out onto main, which stays on 0.13.
- Main's other object-store wrappers compile unchanged: One query grows RSS by >1 GB and never returns it; every query lists the tenant's entire S3 prefix #853 and the Template-map derive and query read metrics: what a cold first query costs #924 fetch counter in the querier, and the RFC 0052 publisher seam in the ingester.
Dependency feature break: one. object_store 0.14's aws feature pulls in aws-lc-rs next to our ring. Commit 2 / fix(serving): pin the client-cert verifier to the ring provider #845 fixes it. Side effects:
- The build is larger (aws-lc-sys).
- One new licence exception is needed for webpki-root-certs; commit 3 adds it.
MSRV: DataFusion 56 declares rust-version = "1.95.0" and pulls in parking_lot 0.13 (also 1.95). The workspace still declares 1.94. See "What must happen at release".

Test results

Latest local run, on 4858ce2 (rc1), rustc 1.99.1:

fmt, clippy (with and without --all-features), cargo deny, cargo-about (regenerated), the AGENTS.md citation check and cargo check --locked for the fuzz crate are all clean.
cargo test --all-features for ourios-parquet, ourios-querier, ourios-df-otel, ourios-ingester and ourios-server passes, except root-only permission tests. Those use chmod/read-only helpers that uid 0 ignores, and they pass on CI's unprivileged runner.

Earlier runs, with rustc/clippy 1.98.1 on f029b944:

	statistics off (this PR, workaround kept)	statistics on (workaround + assertion removed)
`ourios-querier` + `ourios-df-otel` (all RFC 0002 DSL, RFC 0017, RFC 0044, RFC0007.x)	277 passed, 0 failed, 3 ignored (by design)	274 passed, 2 failed, 3 ignored. The count is one lower because the removed assertion test is gone.
Full workspace	1623 passed, 1 failed, 75 ignored	1620 passed, 3 failed, 75 ignored

Earlier findings:

Failing in both configurations (environmental, not caused by this PR): ourios-wal rfc0008_6_persistent_rotation_failure_refuses_appends_for_good. It makes the WAL root read-only with chmod 0555, but my container runs as uid 0, which ignores the mode. It passes on CI's unprivileged runner.
Failing only with statistics on (the B1 / RFC 0044 pruning gate):
- execution::rfc0007_1_pushdown_prunes_row_groups: pruning works (row_groups_pruned: 1, scanned: 1), but bytes_read: 0 fails the "scanned row group reads bytes" assertion.
  - Likely cause: the matching file has one row, so statistics prove every flat column constant. DataFusion substitutes literals and decodes no column bytes.
  - This is a DataFusion behaviour change that the test's wiring guard can't tell apart from a broken metric. I didn't change the test; you need to decide how to treat it.
- rfc0044_body_equality::rfc0044_7_pruning_engages_across_partitions: row_groups_pruned: 0, scanned: 2. #24770 doesn't cover our body == predicate shape.
  - Its proof (stats_alone_unsatisfiable, datasource-parquet/src/opener/mod.rs) only prunes when the predicate reduces to a bare false/NULL literal after statistics constants are substituted.
  - Our template arm also has array_element(separators, k) = … and params terms over list columns, and those never become constants.
  - PhysicalExprSimplifier only does constant evaluation, NOT and unwrap-cast, so it doesn't fold false AND <non-constant>. The predicate never collapses.
  - The substitution has already removed the template_id/body column references, so the pruning predicate has nothing to prune on either. This is still the #24769 shape.
So the statistics flip is not in this PR. It's on branch claude/datafusion-56-collect-statistics-probe (commit edccf1a) for reproduction.

What must happen at release

Swap the git [patch.crates-io] blocks for the crates.io release.
- There are two: the root Cargo.toml and fuzz/Cargo.toml.
- datafusion = "56" and the =60.1.0 arrow/parquet pins are already in place (since 4858ce2), so only the patch blocks go. Refresh both lockfiles and regenerate THIRD-PARTY-LICENSES.md.
- Remove the allow-git entry in deny.toml at the same time.
Raise the workspace rust-version from 1.94 to at least 1.95, or record why it stays. DataFusion 56 declares 1.95, so 1.94 is no longer true. CI's "fuzz toolchain pin" check compares the fuzz nightly against this value.
Drop the collect_statistics = false workaround and its assertion only once RFC0044.7 passes with statistics on. It didn't pass on f029b944 (see above), so shipping DF 56 alone isn't enough.
- Either upstream extends #24770 (for example, a boolean short-circuit in the physical simplifier, or a proof that tolerates non-constant conjuncts under a constant-false AND), or we keep the override.
- Separately, decide whether RFC0007.1's bytes_read > 0 guard is still the right assertion with statistics on.
Update RFC 0021 §3.2a with:
- the upstream status;
- the partial-fix finding;
- the per-file footer read at plan time, which §3.2a already lists as an independent reason to keep collection off.
Close the DataFusion-upgrade part of epic: upgrade DataFusion + Arrow (unblocks object_store 0.14, thrift advisory, arrow-58 decoder dup) #314. Keep the statistics part open until item 3 is resolved.
Squash-merge or re-sign: some merge commits on this branch carry a cloud session's SSH signature that GitHub can't verify.

Upstream timeline

DataFusion 56.0.0 release tracking: apache/datafusion#24461.
- branch-56 was cut on 2026-11-02/03. The "Prepare for 56.0.0 release" commit landed on 2026-11-07, and backports followed.
- 56.0.0-rc1 was tagged at cce6374 (2026-11-12). It is version 56.0.0 on arrow/parquet 60.1.0 and object_store 0.14.2, and its changelog lists #24770. The vote runs at least 72 hours. As of 2026-11-13, crates.io's newest is 55.2.0.
The fix: apache/datafusion#24770 merged on 2026-09-16 as be5a96e. It fixes #24769.
55.2.0 backport: the patch-release tracker apache/datafusion#25758 lists #24769/#24770 with status TBD.
- A backport PR, apache/datafusion#25842, was opened on 2026-09-28 and closed without merging on 2026-09-29. No reason is visible.
- DataFusion 55.2.0 was published to crates.io on 2026-10-07 without the fix. DataFusion 56 is the only vehicle.

Invariants / hazards (AGENTS.md §3–4)

Pillar docs: add verification process spec #1 (footer-skip pruning): unchanged here because the workaround stays. The pruning-gate tests are what showed the upstream fix is partial.
§3.5 (schema): parquet 60 changes no Ourios schema, and the RFC 0021 arrow-upgrade suite passes.
Hazard docs(rfc-0001): add §5 acceptance criteria (drafted → specified) #6 (DSL vs DataFusion surface): no DSL change, and the RFC 0002 suites pass.

Checklist

cargo fmt clean
cargo clippy clean (no new warnings)
Tests added/updated: the maintainer split the decoder-panic test for parquet 60 (28ec00a); I changed no tests.
Docs / CHANGELOG.md updated: at release (RFC 0021)
RFC linked

🤖 Generated with Claude Code

https://claude.ai/code/session_01SBv6mCsBchZiS6CDs9daJg

Patch every datafusion-* crate to apache/datafusion main at f029b944 (carries #24770, the #24769 row-group pruning fix) and move the workspace to the arrow/parquet 60 + object_store 0.14 set that main requires. The workspace checks clean with no source changes. Draft-only: the git patch is swapped for the crates.io release when DataFusion 56 (or a 55.x with the backport) ships. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

TlsSettings::load pins the ServerConfig to ring so the listener does not depend on which rustls providers other crates compile in, but the mTLS path built its WebPkiClientVerifier with the auto-detecting builder. Once a dependency enables rustls/aws-lc-rs next to ring (object_store 0.14's `aws` feature does), that builder panics at startup with "Could not automatically determine the process-level CryptoProvider", failing RFC0030.4 and RFC0030.8. Build the verifier from the same provider. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Claude <[email protected]>

Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Claude <[email protected]>

coderabbitai · 2026-09-27T14:25:59Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…ta license The upstream-revision patch makes cargo-deny see apache/datafusion as a git source, so allow exactly that repository (removed with the patch). object_store 0.14's reqwest 0.13 brings rustls-platform-verifier and with it webpki-root-certs, the same Mozilla CA data as webpki-roots under CDLA-Permissive-2.0; give it the same per-crate exception in deny.toml and about.toml, and regenerate THIRD-PARTY-LICENSES.md with cargo-about 0.9.1. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Claude <[email protected]>

…kspace The fuzz harness is its own workspace and depends on ourios-querier, so the root [patch.crates-io] does not reach it: its lockfile went stale (failing the `--locked` gate in the cargo deny job) and, unpatched, it would pair crates.io DataFusion 55 (arrow 59) with ourios-parquet's arrow 60. Copy the patch block, removed together with the root one, and refresh fuzz/Cargo.lock. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Claude <[email protected]>

Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Claude <[email protected]>

jensholdgaard · 2026-09-27T14:56:22Z

Commit identity for any further commits on this PR (maintainer decision, 2026-09-27). The cloud environment's default git identity is Claude <[email protected]>, so commits made there have been authored and signed off as Claude. The maintainer has decided cloud-agent commits are made as him. Before your next commit, run:

git config user.name "Jens Holdgaard Pedersen"
git config user.email "[email protected]"

Keep using git commit -s, so the sign-off is his, and keep the two attribution trailers (Co-Authored-By: Claude Opus 5.5 <[email protected]> and the Claude-Session: line). Do not rewrite or force-push existing commits to fix their identity: this PR squash-merges, and the squash commit is written with the correct sign-off at merge time.

Resolve the Cargo.lock conflict by keeping this branch's lockfile and letting cargo resolve main's new dependencies onto it. Main's new ourios-ingester and ourios-querier dev-dependencies on object_store 0.13 move to 0.14 so the tree keeps a single ObjectStore trait, matching ourios-parquet and the patched DataFusion. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

Resolve the Cargo.lock conflict (main's v0.11.1 release bump and new dev-dependencies) by keeping this branch's lockfile and letting cargo resolve main's manifests onto it; no new object_store, arrow or DataFusion pins arrived, so no manifest edits were needed. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

jensholdgaard · 2026-10-01T17:38:42Z

cargo clippy failed on 7381003, but not because of this PR.

CI now runs stable Rust 1.99.0 (released 2026-09-28). Its clippy adds the lint assert_is_empty, which rejects assert!(x.is_empty()) in test code. The run stopped after hitting it in:

ourios-miner lib tests: 15 errors, for example crates/ourios-miner/src/upstream.rs:590;
ourios-serving lib tests: 2 errors.

Neither crate is touched by this PR's diff. The same code is on main and will fail the same way once main's CI runs on 1.99. My local check passed only because this container still had 1.98.1.

There's no fix to port yet: no commit on main and no open PR addresses it. Two options for whoever fixes it on main, following the precedent of #839 for clippy 1.98:

rewrite each site as clippy suggests (assert_eq!(x, [])), which shows the value when a test fails;
add a crate-level #[allow(clippy::assert_is_empty)] in the test modules, with a // justification.

Once a fix lands on main, I'll merge it into this branch. I won't re-run the job, because the failure is deterministic.

Separately, ourios-ingester's compactor::graph_tests::rfc0048_4_completion_event_carries_the_registry_name is flaky on main too: it failed in 2 of 10 local full-suite runs on main and passes on its own. If it shows up here, it's also not this PR's.

Generated by Claude Code

Brings in the clippy 1.99 fix (#901) that this branch's CI needs, plus main's v0.11.2 changes. Merges cleanly; both lockfiles stay current. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

Main's dependency pinning (#902) turned the ourios-ingester and ourios-querier object_store dev-dependencies into exact `=0.13.2` pins, conflicting with this branch's 0.14 bump. Keep main's exact-pin style at this branch's locked version, `=0.14.2`, so the tree still has a single ObjectStore trait; take main's other pins and the 1.99.0 toolchain pin (#906) as they are. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

Main's durable template-id allocation (RFC 0059, #911) moves ourios-ingester's object_store and async-trait from dev- to regular dependencies, conflicting with this branch's object_store bump. Take main's layout and keep object_store at this branch's `=0.14.2`, so the tree still has a single ObjectStore trait; the RFC 0059 code compiles unchanged against object_store 0.14. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

Main's RFC0059.6 SIGTERM test adds an exact `arrow-array = "=59.3.0"` dev-dependency to ourios-server next to its parquet pin. Take main's lines and keep both at this branch's arrow/parquet `=60.0.0`, so the tree still carries one arrow and one parquet. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

Keep object_store at =0.14.2 in the querier dev-dependencies, and pass the new `GetResult::extensions` field through `MeteredStore::get_opts` so HTTP response extensions survive the metering wrapper. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y2NNRu7nTGmcK4BoWUy3s2 Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

jensholdgaard · 2026-11-10T09:33:27Z

cargo test (1/2) failed on cd969fb in a raft test from main, not because of this PR.

ourios-ingester replicated::forward::tests::a_follower_names_the_leader_and_a_wrong_group_is_not_hosted
panicked at .../replicated/forward/tests.rs:128:5
  left: Err(NotLeader { leader: None })
 right: Err(NotLeader { leader: Some("00000000-0000-0000-0000-000000000001") })

Why it isn't this PR's. crates/ourios-ingester/src/replicated/forward/tests.rs is byte-identical to main, and the code it exercises doesn't depend on DataFusion or arrow.

Cause. This is the race #1243 fixed in the sibling test a_forward_to_the_leader_commits_and_one_to_a_follower_names_it. The follower answers NotLeader with the leader's node from its membership, and it can answer before it follows the leader or holds that membership entry. #1243 added a wait to the sibling test only; this test asks the follower right after ready_leader without one.

Proposed fix for main. Move #1243's wait into a helper and call it in both tests:

/// Wait until `follower` follows `leader` and holds its node: the
/// `NotLeader` hint is the leader's node from the follower's membership,
/// which a heartbeat can precede.
async fn follows(cluster: &Cluster, follower: usize, leader: usize) { /* #1243's wait */ }

Then a_follower_names_the_leader_and_a_wrong_group_is_not_hosted calls follows(&cluster, follower, leader).await before forwarding to the follower.

Local checks:

cargo fmt --check and cargo clippy -p ourios-ingester --all-targets --all-features -D warnings are clean.
cargo test -p ourios-ingester --all-features --lib replicated passed 25 of 26 runs with the patch. The one failure came from a batch whose output I didn't capture, so I can't say which test it was.

No fix exists on main or in an open PR yet, so I've re-run the failed job once.

Generated by Claude Code

Copilot

🔵 Needs a closer look

The unreleased DataFusion query/storage upgrade requires maintainer sign-off and release-time CAS, pruning, and benchmark validation that could not be performed in this review environment.

No open issues 🎉

_{Review effort: Balanced}

Repoint the [patch.crates-io] DataFusion crates (root and fuzz) from the upstream main revision to 56.0.0-rc1 (cce6374), which carries the #24769 row-group pruning fix (#24770). The patched crates now report 56.0.0, so the workspace asks for datafusion "56", and the arrow and parquet pins move to the 60.1.0 that release builds on. object_store stays 0.14.2. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SBv6mCsBchZiS6CDs9daJg Signed-off-by: Jens Holdgaard Pedersen <[email protected]>

jensholdgaard · 2026-11-13T17:59:07Z

DataFusion 56.0.0-rc1 is tagged upstream. This branch now builds against it, as of 4858ce2.

56.0.0-rc1 points at branch-56 commit cce6374 (2026-11-12). It is version 56.0.0 on arrow/parquet 60.1.0 and object_store 0.14.2, and its changelog includes #24770, the #24769 row-group pruning fix. crates.io's newest is still 55.2.0.

What changed in this commit:

DataFusion patch: the [patch.crates-io] DataFusion crates in both the root and fuzz/ Cargo.toml now use rev = "cce6374…" (the rc1 commit) instead of the old main revision f029b94. I updated the comments to match.
Version requirement: the patched crates now report 56.0.0, so ourios-querier and ourios-df-otel ask for datafusion = "56". A "55" requirement would skip the patch.
arrow/parquet pins: the =60.0.0 pins in bench, ingester, parquet, querier and server move to =60.1.0. object_store stays at =0.14.2.
Lockfiles and licences: both lockfiles are re-resolved with no "patch was not used" warnings, and THIRD-PARTY-LICENSES.md is regenerated.

Local checks:

Passed: cargo fmt --check, both clippy runs (--all-features and plain --workspace), cargo deny check (advisories, bans, licences and sources), cargo about regenerated, the AGENTS.md citation check, and cargo check --locked for the fuzz crate.
Tests: cargo test --all-features for ourios-parquet, ourios-querier, ourios-df-otel, ourios-ingester and ourios-server failed only the known chmod tests that fail when run as root. Those are querier's rfc0033_11, ingester's 7 compactor/reclaim tests and rfc0059_6; all pass on CI's unprivileged runner. Every other test passed, including the logs DSL and the decoder-panic tests.

One thing to decide at release:

MSRV: DataFusion 56 declares rust-version = "1.95.0", and it pulls in parking_lot 0.13, which also needs 1.95. The workspace still says rust-version = "1.94".
What's affected: CI and the images build on the pinned 1.99.1, so nothing breaks today, but the declared MSRV is no longer true. CI's "fuzz toolchain pin" check also compares the fuzz nightly against this value.
I haven't bumped it: that's an MSRV decision, and I didn't want to make it in a draft.

Next: the release vote stays open at least 72 hours. When 56.0.0 is on crates.io, the steps under "What must happen at release" in the PR body apply. I'll notify you then. The PR stays a draft.

Generated by Claude Code

Copilot

🔵 Needs a closer look

The coordinated RC-backed upgrade still requires maintainer confirmation of the final dependency graph and RFC 0021 phase-2b gate, plus S3 CAS and pruning/performance validation not established by the supplied local results.

No open issues 🎉

_{Review effort: Balanced}

codescene-access

Gates Failed
Enforce advisory code health rules (1 file with Code Duplication)

Our agent can fix these. Install it.

Gates Passed
5 Quality Gates Passed

Reason for failure

Enforce advisory code health rules	Violations	Code Health Impact
decoder_panics.rs	1 advisory rule	10.00 → 9.39	Suppress

See analysis details in CodeScene

Quality Gate Profile: Pay Down Tech Debt
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.

jensholdgaard · 2026-11-14T00:08:47Z

AI-assisted reply (Claude Code)

rc1 with collect_statistics = true (tested 2026-11-14): the querier and df-otel suites pass, except the stats-off assertion and two pruning tests:

RFC0044.7: the non-candidate partition is not pruned. A minimal DataFusion-only reproduction shows this happens when the predicate has a term that can't be used for pruning, in our case array_element(separators, n) = '…'. #24770 fixed the case without such a term, and 55.2.0 misses both. Reported upstream as Row group pruning with collect_statistics = true is still lost when the predicate has a non-prunable term (follow-up to #24769) apache/datafusion#27415. It isn't a 56 regression, so it shouldn't block the vote.
RFC0007.1: pruning works, but bytes_read is 0. The matched file's template_id is constant, so the count is answered without reading column data. Only the test's "reads bytes" guard fails, and it needs rewording once statistics go back on.

So collect_statistics stays off on this branch until #27415 is fixed. Moving to 56.0.0 can still go ahead once it's on crates.io.

jensholdgaard · 2026-11-14T00:15:41Z

AI-assisted reply (Claude Code)

A second, stronger reason to keep collect_statistics off: apache/datafusion#26490 (open, filed by someone else on 2026-10-19).

The same statistics-to-constant shortcut returns wrong values. Parquet min/max statistics leave out NaN, so a float column holding one value plus NaNs looks constant, and DataFusion returns that value for the NaN rows. Only collect_statistics = false avoids it.

Ourios writes OTLP DoubleValues, NaN included, into its promoted Float64 attribute columns (crates/ourios-parquet/src/promoted.rs:214, RFC 0042). With statistics on, those rows would read back wrong, which breaks OTLP fidelity. The fix PRs upstream (#27283, #27290) are open, and #26494 was closed unmerged, so 56.0.0-rc1 most likely still has it.

The gate for switching statistics back on is now #27415 and #26490 both fixed in a release, plus a NaN-in-a-promoted-column test of our own before flipping it.

jensholdgaard and others added 3 commits September 27, 2026 15:20