Upgrade to DataFusion 55.1 and Arrow 59.3 by krinart · Pull Request #14612 · spiceai/spiceai

krinart · 2026-09-30T21:55:33Z

📝 Summary

Upgrades Spice to DataFusion 55.1 and Arrow 59.3. Every forked dependency moves to its DataFusion 55 / Arrow 59 line, and each fork is pinned at the commit its PR merged into its version branch (fork PRs below).

Statistics flow through DataFusion 55's StatisticsContext. On 55, the deprecated partition_statistics returns no statistics for built-in plans, so wrappers, optimizer rules, join sizing and Flight batch sizing are migrated to keep seeing real row counts.
Plan wrappers that delegate a projection swap now re-root the projection on their inner plan. DataFusion 55's projection pushdown reads the projection's input as the node being swapped with, and delegating it unchanged made plans grow without bound.
Arrow 59 refuses a MAP whose entries field is declared nullable. Flight SQL reads repair the declared schema before decoding, via a new arrow-rs fork API, and legacy IPC data decodes the map as a list and restores it.
The DuckDB rendering of regexp_count no longer coalesces NULL to 0, so accelerated and local evaluation agree under DataFusion 55's NULL-propagating regexp_count.
Ballista 55 (upstream 55.0.0-rc1): vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model.
Wrong-result fixes found during the upgrade, each with a test that fails without it:
- SUM(col) over an inner or outer join was answered from statistics with the whole input table's sum. DataFusion 55's AggregateStatistics read join statistics that carried the input sums through. Fixed in the DataFusion fork (fix(physical-plan): drop input sums from inner and outer join statistics datafusion#239).
- A binary literal pushed to DuckDB (X'ff') was read as text, returning the wrong rows. Such filters now stay local.
- Repairing a nullable map-entries schema re-numbered dictionary ids, so a Flight producer with different ids decoded columns against each other's dictionaries. The repair now patches the schema message in place.
datafusion-spark 55: trunk's collision table (Spark functions are registered over DataFusion built-ins via a deny-list: 7 live collisions, factorial unusable and concat silently Spark's #14361) decides pow/power, atan2 and concat_ws as Keep::BuiltIn, so each answers as it did on 54. hypot, monthname, quote and weekday are not registered, because they would be new SQL surface.
Cayenne keeps small Vortex scans whole on the main query path. DataFusion 55 lowered repartition_file_min_size from 10 MiB to 1 MiB (feat: lower repartition_file_min_size default from 10 MiB to 1 MiB apache/datafusion#22439), which split 1–10 MiB dimension tables into target_partitions ranges, each paying a Vortex footer open. The main scan now opts out below 10 MiB, as its internal scans already do below 256 MiB. Parquet listing scans keep the new default.

🔗 Related

🚨 Breaking Changes

These come from DataFusion 55 semantics, not from new Spice behavior:

regexp_count returns NULL for a NULL input or pattern (it returned 0), locally and on DuckDB acceleration.
round on an integer argument returns the integer type (it returned Float64); fix: Preserve integer values in round() for large Int64 and UInt64 inputs apache/datafusion#22697.
SQL = on floats compares by IEEE value, so -0.0 = 0.0 is true.
Some EXPLAIN text changes. For example, decimal literals print as Decimal128(123.45,10,2).
Cluster mode: schedulers and executors must run the same version. Ballista 55 adds a protocol-version handshake, and TaskCancelInfo drops partition_id.

👀 Notes for Reviewers

docs/dev/fork_patches.md records every pin as landed. duckdb-rs is the one exception. Move duckdb-rs 1.4.4 to Arrow 59 duckdb-rs#49 was squash-merged, so it stays at 8ee43073, the head of the protected spiceai-1.4.4-patches-2 branch. datafusion-table-providers pins exactly that revision, so the two move to spiceai-1.4.4 together in a follow-up, which also brings duckdb-rs User friendly: restructure directory #50 and Get Brew PR ready #51.
The full upstream DataFusion suite was run on the fork against spiceai-54. No sqllogictest record returns different rows. The unit-test failures that remain are the ones spiceai-54 already has: Spice patches changing plan or SQL text, and a backtrace-feature artifact.

Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge.

- Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto).

…tches The DataFusion 55 line of the fork is spiceai-55-patches.

d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit.

… evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern.

DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne.

- Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None.

The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change.

github-actions · 2026-09-30T21:55:47Z

✅ Pull with Spice Passed

Passing checks:

✅ Title meets minimum length requirement (10 characters)
✅ No banned labels detected
✅ Has a label from required category kind/
✅ Has a label from required category area/
✅ Has at least one assignee: krinart

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…-att-2

Copilot

Copilot review overview

🔵 Needs a closer look

JSONL split alignment can read one byte beyond files whose final record has no trailing newline.

Review effort: Balanced
Findings: None

Resolved since last review (2)

Route get_schema fallback through nullable-map schema repair
Correct summary and document decimal power type/result breaking change

krinart added 8 commits September 30, 2026 02:25

ci: validate the datafusion-table-providers pin against spiceai-55-pa…

d65aa29

…tches The DataFusion 55 line of the fork is spiceai-55-patches.

revert: drop the unfinished guard tests d65aa29 picked up by accident

b584210

d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit.

test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test

dd52180

DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne.

Copilot AI balanced review requested due to automatic review settings September 30, 2026 21:55

github-actions Bot added area/ci area/config area/docs area/tests kind/dependencies Pull requests that update a dependency file labels Sep 30, 2026

krinart added the breaking change label Sep 30, 2026

Copilot started reviewing on behalf of krinart September 30, 2026 21:56

View session

Copilot AI reviewed Sep 30, 2026

lukekim previously approved these changes Oct 5, 2026

View reviewed changes

lukekim enabled auto-merge October 5, 2026 19:08

lukekim added this pull request to the merge queue Oct 5, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026

sgrebnov added this pull request to the merge queue Oct 5, 2026

krinart added 2 commits October 5, 2026 16:06

Merge remote-tracking branch 'origin/trunk' into viktor/datafusion-55…

e5463e0

…-att-2

Fix lint

6c8d05a

krinart removed this pull request from the merge queue due to a manual request Oct 5, 2026

Copilot AI balanced review requested due to automatic review settings October 5, 2026 23:12

krinart dismissed lukekim’s stale review via 6c8d05a October 5, 2026 23:12

Copilot started reviewing on behalf of krinart October 5, 2026 23:12

View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

sgrebnov approved these changes Oct 5, 2026

View reviewed changes

spiceemma mentioned this pull request Oct 6, 2026

test(sqlite): guard the Date32 literal cast through the SQLite acceleration's own scan (fixes #14491) #14703

Merged

2 tasks

This was referenced Oct 6, 2026

fix(listing): keep a partition predicate as a residual filter on the _location fast path #14790

Merged

fix(runtime): serve an existing acceleration while its source is unavailable (fixes #14610) #14624

Merged

spiceemma mentioned this pull request Oct 6, 2026

docs(sql): regexp_count returns NULL for a NULL argument, locally and on DuckDB spiceai/docs#2310

Merged

grokspice mentioned this pull request Oct 6, 2026

ci(codeql): don't fail the SARIF upload when the merge queue already deleted its ref #14793

Merged

8 tasks done

krinart mentioned this pull request Oct 9, 2026

docs: document behavior that changed in v2.4.0 spiceai/docs#2332

Merged

Jeadie mentioned this pull request Oct 12, 2026

fix: Update Search integration test snapshots #15050

Merged

phillipleblanc mentioned this pull request Oct 13, 2026

fix(vortex): keep casts Vortex cannot evaluate out of the scan #15096

Merged

Navigation Menu

Repository navigation

Upgrade to DataFusion 55.1 and Arrow 59.3 - #14612

Upgrade to DataFusion 55.1 and Arrow 59.3#14612
krinart merged 52 commits into
trunkfrom
viktor/datafusion-55-att-2

krinart commented Sep 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

krinart commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

🔗 Related

🚨 Breaking Changes

👀 Notes for Reviewers

Uh oh!

github-actions Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Pull with Spice Passed

Passing checks:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

krinart commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Sep 30, 2026 •

edited

Loading