Repository navigation
Merge upstream Ballista (DataFusion 55) into spiceai-55 - #68
Merged
Merged
Conversation
Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.7.18 to 0.7.19. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-util-0.7.18...tokio-util-0.7.19) --- updated-dependencies: - dependency-name: tokio-util dependency-version: 0.7.19 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…K sorted disjoint (apache#2169) * feat(core): add OrderedRangeRepartitionExec — N sorted overlapping → K sorted disjoint Sibling of `UnorderedRangeRepartitionExec` (apache#2123). Same discovery mechanism (walk child subtree for matching `RuntimeStatsExec`, snapshot T-Digest, compute quantile cuts via `range_repartition_common::discover_cuts`); different execution model. Semantics: takes N locally-sorted input partitions whose value ranges may overlap; produces K range-disjoint output partitions where each output is fully sorted on the routing expression. Concatenating the K outputs in order yields a globally-sorted stream — exactly what Stage 2 of the parallel-window path needs to feed into Stage 3's BWAG. Execution model — N × K channels + K k-way merges: * N scatter tasks. Each reads one sorted input, calls `split_batch_by_range` on every batch, and forwards sub-batches to K per-input senders (one per output partition). * K output-partition streams. Each merges N sub-streams via DataFusion's `StreamingMergeBuilder` — SPM internals, no custom heap machinery here. * Channels: bounded `mpsc::channel(2)` per input × output for symmetric backpressure with the Unordered variant. Total buffered at rest ≤ N × K × 2 batches. Constructor requires `input.output_ordering()` to lead with the routing expression — otherwise the merger would produce garbled output. `PlanProperties::eq_properties` advertises each output partition as sorted on `order_by`, so downstream operators (BWAG, HaloDrop) can rely on that claim without an intervening `SortExec`. Proto + serde follow the Unordered pattern (`uint32 output_partitions` only; boundaries are runtime-discovered, stateless w.r.t. serde). Eight end-to-end tests: conservation across two overlapping inputs, each output sorted, outputs range-disjoint, no-stats fallback, constructor validation (empty order_by, unsorted input, mismatched sort key), and the double-execute guard. One serde roundtrip test on top. Drift from source branch (`brent/h2o-window-benchmarks`): * `RuntimeStatsExec` now rejects nullable routing expressions at construction (T-Digest has no NULL slot; tightened during apache#2123's upstreaming). Original tests declared `v2` as nullable and typed the batch builder as `Vec<Option<f64>>` but never populated a `None`. Tightened schema + helper types to non-nullable `f64`; will loosen again when we swap T-Digest for KLL and NULL becomes a first-class routing key. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * review: address review-preference audit for OrderedRangeRepartitionExec Mirrors the audit Phillip asked for on apache#2123 (commit a281667 on the unordered sibling). Preemptive polish before opening the PR for review. - Implement all 6 algebraic ExecutionPlan methods explicitly (required_input_ distribution, required_input_ordering, maintains_input_order, benefits_from_ input_partitioning, partition_statistics, cardinality_effect), each with a docstring justifying the value. Two differ materially from the unordered sibling: * required_input_ordering returns Some(order_by): the child MUST be sorted on the routing expression — the per-output StreamingMerge assumes each of its N sub-streams is locally sorted. Declaring the requirement lets the optimizer see it and skip a redundant SortExec above us. * maintains_input_order returns true: each output is a k-way merge across N sorted sub-streams filtered to a value range, so the relative order within each output faithfully preserves the child's sort. That's the whole point (contrast Unordered, which returns false because scattering breaks order). - Reject nullable routing expressions in try_new. Defense-in-depth against the no-stats-pair fallback path where discovery would silently fall back to single-bucket. TODO marker points at the KLL swap that lifts the restriction. The sibling RuntimeStatsExec already enforces the same invariant at plan-assembly time. - Add try_new_rejects_nullable_routing_key covering the new invariant. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * review: surface scatter-task panics + set Eager/Cooperative on ORRE Ports apache#2123's review-fix commits (6b300df + 422ee6a) to the ordered sibling. 1. `scatter_input_partition` was spawned as a bare `tokio::spawn` whose `JoinHandle` was discarded. A panic inside the body silently dropped the sender, and downstream consumers saw a clean EOF while data was lost. Wrap it in `SpawnedTask::spawn(guarded_scatter(...))` (helper already shared via `range_repartition_common` — added for apache#2123 and consumed here verbatim). Hold the wait-tasks in `DispatchState._drop_helper` so dropping the exec aborts all N scatter tasks. Refactored the scatter body to return `Result<()>` instead of internally calling `broadcast_error` — guarded_scatter handles the broadcast on error/panic. 2. `PlanProperties::new` defaults to `Lazy` / `NonCooperative`; ORRE eagerly drives its inputs from the moment `execute()` is called and yields at per-output channel sends, so set `EvaluationType::Eager` + `SchedulingType::Cooperative` — same combination DataFusion's `RepartitionExec` uses. 3. New `scatter_task_panic_surfaces_as_error` test asserts every output partition surfaces the panic as `Err(..)` with the original message preserved. `OrderedRangeRepartitionExec::try_new` rejects inputs without a declared ordering, so extend `PanickingSourceExec` with a `with_ordering(order_by)` constructor that lets the panic-source satisfy the ordering-claim precondition. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
* kll: seed KllSketch<T: Ord> with rank-below-capacity test New module ballista_core::kll. Trivial Vec-backed impl (no compaction yet) that satisfies the base case: for n <= capacity, rank(x) matches an exact linear scan. Test is red-green with u32; the same shape against OwnedRow is next. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: mirror the rank test through arrow-row OwnedRow Same 10 items, same probe, same expected rank — but the items live as OwnedRow bytes produced by RowConverter, exercising the production item type. No sketch-side changes: KllSketch<T: Ord> composes with OwnedRow via its Ord impl (lex byte compare over the encoded row). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: add null-bearing arrow-row rank test Adds a third test that stitches NULLs into the input array and probes with a non-null value. Under SortField's default (ASC NULLS FIRST), encoded NULLs sort strictly below any encoded Some(_), so rank(Some(5)) picks up the null count on top of the four values below 5. Sketch unchanged: OwnedRow's Ord impl carries the policy. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: add multi-column arrow-row rank test Fourth honesty test: two-column key (a UInt32, b UInt32), lex ASC/ASC. Same 6 stream rows produce OwnedRow items; the probe (2, 10) picks up (1,10), (1,20), and (2,5). Excluded: (2,15) same a with larger b, and the (3,*) pair with larger a. Cements that the Vec<Option<dyn Ord>> logical model is fully realized as `T = OwnedRow` — arbitrary column count and per-column SortOptions flow through the encoding, sketch stays generic. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: document the sketch and its role in Ballista Adds module-level rustdoc: why this sketch exists here (the scheduler needs multi-column, nullable, mixed-direction sort keys for global partition points, which composes as T = OwnedRow), the compactor-stack layout with per-level weights 2^h, the compaction procedure (sort, fair coin, keep half, promote), the weighted rank formula, and the role of k. Includes a link to the Karnin-Lang-Liberty FOCS 2016 paper. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: earn the K — compactor stack with weighted rank Replaces the single Vec<T> with a Vec<Vec<T>> compactor stack indexed by level. When level h fills to k, its contents are sorted, halved by a fair coin flip (odd or even indices), and the surviving k/2 items are promoted to level h+1 with double weight; the cascade continues upward while any level remains full. Rank sums 2^h · (count-below-x) over levels. Constructor becomes new(k) plus with_seed(k, seed) for deterministic tests, backed by an owned StdRng — the coin flip cannot be non-reproducible. The zero-arg new() and Default are gone; k has no natural default. The four earlier tests move to with_seed(1024, 42), below capacity and thus unchanged in behavior. New test locks two exact invariants that hold regardless of coin flips: for a stream of 1..=32 with k=4, rank(33) == 32 (total-weight conservation across cascades) and rank(1) == 0 (nothing below the min). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: document deferred ownership as a future optimization The current per-item insert API materializes each Arrow Row<'_> into an OwnedRow on the way in — most of those allocations are later discarded by cascading compactions and are effectively wasted. Documents the tradeoff and sketches the shape of the future optimization (a batch-oriented ingest API that holds borrowed rows in level 0 for the duration of a batch), so the choice is visible before real callers accumulate against the current shape. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: shrinking-k per level, bounded total space Ports the level-capacity formula from Apache DataSketches (kll_helper): each level's capacity is ceil(k · (2/3)^depth) floored at m = 8, where depth counts down from the top. The top level always has capacity k, and every level below it is 2/3 the size of the one above — so the geometric sum converges to ~3k and total retained items is bounded constant in n rather than growing O(k · log(n/k)). Compaction is now a fixed-point loop: find the lowest level whose count exceeds its (dynamically shrinking) capacity, compact it, and loop until none is over. This handles the case where adding a new top level shrinks the effective capacities of every existing level, and the same driver will serve merge (multiple levels may be over capacity at once, unlike after an insert). Odd-length compaction is not skipped: the smallest item is retained at the current level with unchanged weight, and the even remainder is sorted, coin-flipped, and halved. Necessary because the (2/3) formula produces odd capacities, and any odd count on any level would otherwise drift the total-weight invariant by one item's worth per compaction. New test inserts 10K items with k=64 and verifies total retained items stays under the shrinking capacity sum (much tighter than the naive k · num_levels bound), while the mass invariants still hold. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: add quantile query with multi-level HOF exercise Introduces KllSketch::quantile(q) — returns the smallest retained item whose cumulative weight (across all levels, weighted by 2^h) reaches q · total_weight. Non-interpolated: the returned reference is always to an item actually present in the sketch, matching the shape the scheduler needs for partition-boundary picking. Empty sketch returns None; q is clamped to [0, 1]. Three tests, layered: - quantile_on_empty_returns_none: the None branch. - quantile_is_exact_below_capacity: n < k, single-level sketch, exact 0.0/0.5/1.0 answers. - quantile_range_and_rank_roundtrip_under_compactions: n = 10K with k = 64 to force multi-level state; asserts every returned quantile lies in the input range and that rank(quantile(q)) tracks q · n within KLL's normalized error bound (~0.23 at k = 64). This third test exercises the multi-weight cumulative walk and the sort-across-levels step that the exact test does not touch. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: track true min/max outside the compactor stack Compactions can coin-flip the true stream extremes out of the compactor levels, so quantile(0.0) and quantile(1.0) drifted from the input's actual min and max — an unacceptable shape for partition-boundary picks. Matches Apache DataSketches' separately maintained min_item_/max_item_ pattern. Adds a Clone bound to T (satisfied by u32 and OwnedRow, the two item types the sketch is used with today) and two Option<T> fields updated on insert; the clone fires only when x is a new extreme, which is rare after warmup. quantile(0.0)/quantile(1.0) now short-circuit to those fields before touching the compactors. The new test inserts 10K items with k=64 (deep cascades) and asserts quantile(0.0) == Some(&1) and quantile(1.0) == Some(&10_000). Without the fix it returned Some(&3) at the low end — direct evidence the true minimum had been coin-flipped away. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: merge two sketches by extending same-height compactors New KllSketch::merge(&mut self, other: Self): folds one sketch into another and preserves the standard invariants. Same-height compactors concatenate (items at level h from either sketch carry weight 2^h), extremes cross-fold via explicit min/max on T (dodging the trap that Rust's Option::cmp puts None below any Some), and compact_all walks the full stack afterwards because a merge can leave several levels over their shrinking capacities simultaneously — which is exactly why compact_all is a fixed-point loop instead of a cascade from level 0. Mismatched k is a caller error and caught with debug_assert; tracking a min-of-observed-k like DataSketches is future work. Also documents the "levels are not sorted between operations" choice as a legitimate perf follow-up: DataSketches carries per-level sortedness plus an is_level_zero_sorted_ flag to skip re-sorts and to merge via linear-time merge-sort; we re-sort at every compaction, trading ~log(k) comparisons per amortized insert for shape simplicity. Test: merge two disjoint u32 streams (1..=32 and 33..=64) into one sketch, then verify combined mass (rank(65) == 64), below-min (rank(1) == 0), and that quantile(0.0)/quantile(1.0) return the folded extremes. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: add ASF license header to satisfy RAT check RAT (Release Audit Tool) requires every source file to carry the standard Apache 2.0 header; PR CI flagged its absence. Copied the verbatim boilerplate used by every other .rs file in the crate. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * kll: test merge across differing compactor stack heights The existing merge test used two same-sized streams, so both sketches end up with roughly the same number of levels — never exercising the height-asymmetry cases that matter in production (executors observe wildly different item counts before their sketches are combined). Two new tests, each in one orientation: merge_grows_stack_when_other_is_taller — small sketch absorbs a large one. The loop hits `push(Vec::new())` on every iteration past small's original top; asserts the stack grew to at least large's height so no promoted item is silently lost. merge_preserves_upper_levels_when_other_is_shorter — large sketch absorbs a small one. The loop terminates before large's upper levels; asserts they weren't dropped and that mass/extremes at the high end are intact. Both also assert the standard invariants (mass, below-min, exact min/max), so a bug that only shows up when heights differ would surface here rather than in the symmetric test. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Port the 54.0.0 changelog from branch-54 to main and add it to the changelog index toctree.
…2191) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.85.0 to 2.85.2. - [Release notes](https://github.com/taiki-e/install-action/releases) - [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md) - [Commits](taiki-e/install-action@7572810...41049aa) --- updated-dependencies: - dependency-name: taiki-e/install-action dependency-version: 2.85.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.9.0 to 1.10.1. - [Release notes](https://github.com/smithy-lang/smithy-rs/releases) - [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/smithy-lang/smithy-rs/commits) --- updated-dependencies: - dependency-name: aws-config dependency-version: 1.10.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ctor](https://github.com/mmastrac/linktime) from 1.0.10 to 1.0.11. - [Release notes](https://github.com/mmastrac/linktime/releases) - [Changelog](https://github.com/mmastrac/linktime/blob/master/CHANGELOG.md) - [Commits](mmastrac/linktime@ctor-1.0.10...ctor-1.0.11) --- updated-dependencies: - dependency-name: ctor dependency-version: 1.0.11 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ates (apache#2133) Bumps the github-codeql group with 2 updates in the / directory: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.1 to 4.37.3 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@7188fc3...e4fba86) Updates `github/codeql-action/analyze` from 4.37.1 to 4.37.3 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@7188fc3...e4fba86) --- updated-dependencies: - dependency-name: github/codeql-action/analyze dependency-version: 4.37.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-codeql - dependency-name: github/codeql-action/init dependency-version: 4.37.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-codeql ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 8.3.2 to 9.0.0. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@11f9893...c771a70) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Martin Grigorov <[email protected]>
…; log merged quantile cuts per stage (apache#2175) * proto: add RuntimeStatsReport wire format on SuccessfulTask Groundwork for shipping RuntimeStatsExec observations (row counts + quantile sketches) back to the scheduler for global-cut selection. Wire format only; behavior unchanged. The executor still emits `runtime_stats: vec![]` at every callsite. Slice B populates it. Proto additions (`ballista.proto`): * `SuccessfulTask.runtime_stats: repeated RuntimeStatsReport`. One entry per RuntimeStatsExec that is still valid at the plan's output (i.e., reachable from the top through distribution- preserving nodes only, per `preserves_distribution` in range_repartition_common). Pre-repartition stats stay local to the executor (used only for the repartitioner's own approximate routing) — the walker's whitelist naturally excludes them. * `RuntimeStatsReport { order_by, partitions }`. `order_by` tags the routing expression so the scheduler groups sketches across tasks that were sampling the same expression. `partitions` carries one entry per observed partition — pre-repartition one per input partition, post-repartition one per output sub- partition; the operator's position determines the interpretation. * `RuntimeStatsPartitionEntry { partition_id, row_count, optional sketch }`. A future `optional MinMaxState min_max` slot is called out as a TODO — the eventual lighter mode for post- repartition bin-pack where full T-Digests are overkill. Note: `QuantileSketchState` and the `sketch_to_proto` / `sketch_from_proto` helpers already landed in main with apache#2094 and are reused here without change. Constructor plumbing: * Five existing `SuccessfulTask { ... }` sites populated with `runtime_stats: vec![]` — the real executor path (executor/src/lib.rs), one scheduler-server integration-test path, and three scheduler test fixtures. Verification: * `cargo build --workspace` — clean. * `cargo test -p ballista-core --lib` — 166 pass (existing wire tests over `sketch_to_proto` / `sketch_from_proto` cover the payload; prost codegen refuses field-number clashes at build time so no new roundtrip test is warranted for a purely structural addition). * `cargo test -p ballista-scheduler --lib` — 269 pass. * `cargo test -p ballista-executor --lib` — 53 pass. * `cargo clippy --all-targets` — clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> executor: ship RuntimeStatsReports to scheduler; log on arrival End-to-end path for stage-1 quantile sketches — extract on the executor at task completion, transport in `SuccessfulTask.runtime_stats` (landed in the previous commit), and log a per-report summary on the scheduler as proof-of-life. Per-stage accumulation + merged-quantile-cut logging comes next. # Executor extraction (runtime_stats.rs) `collect_reports(plan)` walks the plan through the `preserves_distribution` whitelist and collects one `RuntimeStatsReport` per reachable `RuntimeStatsExec`. Stats-taps sitting below any distribution-changing operator (e.g. `UnorderedRangeRepartitionExec`) are excluded automatically because the walker stops at that boundary; their sketches describe data the repartitioner then routed away and are no longer meaningful at the plan's output. Each emitted report carries the operator's `order_by` (serialised as a `PhysicalSortExprNode` — the wire tag the scheduler will group on) and one `RuntimeStatsPartitionEntry` per partition slot with `{partition_id, row_count, optional sketch}`. Empty sketches are elided (`sketch: None`) so bin-pack space is proportional to actual samples, not slot count. `sketch_to_proto` / `sketch_from_proto`, `partition_count()`, and the widened `preserves_distribution` whitelist already exist in main — introduced with the RuntimeStatsExec / URRE PRs. This commit reuses them. # Trait plumbing (execution_engine.rs) `QueryStageExecutor::collect_runtime_stats_reports()` — default returns empty. `DefaultQueryStageExec` overrides to call `collect_reports` against whichever `ShuffleWriterVariant` it holds (`Passthrough` or `Sort`). Serialisation errors are logged and the report dropped rather than failing the task; the task's data was already produced correctly, telemetry loss shouldn't tank the query. # Transport (executor lib + call sites) `as_task_status` gains a `runtime_stats: Vec<RuntimeStatsReport>` parameter (marked `#[allow(clippy::too_many_arguments)]`) and populates `SuccessfulTask.runtime_stats` with it. Both executor call-sites (`execution_loop.rs`, `executor_server.rs`) call `collect_runtime_stats_reports()` on the query-stage executor and thread the result through. Failure paths pass `vec![]`. # Scheduler receive (task_manager.rs) `TaskManager::update_task_statuses` calls a new `log_runtime_stats_arrival` helper on each status before batching. Logged at `debug!` — RUST_LOG can promote it locally during verification. Fields: executor / job / stage / task + per-report summary (order_by_len, partition count, non-empty partitions, total rows, sketches). # Tests Four new walker tests in `execution_plans::runtime_stats::collect_tests`: * `collect_reports_finds_stats_and_ships_sketch` — stats at plan root, populated sketch survives `sketch_from_proto`, row_count matches drained data. * `collect_reports_row_count_only_emits_report_without_sketch` — row-count-only mode emits a report but no sketch payload. * `collect_reports_descends_through_whitelisted_op` — a `BufferExec` intermediary does not block the walker. * `collect_reports_stops_at_sort_that_collapses_partitions` — N→1 SortExec is explicitly excluded from the whitelist; the walker respects that. # Verification * `cargo test -p ballista-core --lib` — 170 pass (was 166). * `cargo test -p ballista-scheduler --lib` — 269 pass, 2 ignored. * `cargo test -p ballista-executor --lib` — 53 pass. * `cargo clippy --all-targets` — clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> scheduler: accumulate runtime-stats reports per stage; log merged cuts Barrier 1.5 prep. On the scheduler, each stage-attempt now accumulates the `RuntimeStatsReport`s that arrive with successful task statuses. When the stage transitions to Successful, the reports are grouped by `order_by` wire tag, sketches merged via `TDigest::merge_digests`, and one debug! line emitted per group with the `K-1` quantile cuts a globally-informed router would have used for `K` output partitions. `K` is read from the reports themselves — every report carries one partition entry per repartitioner-output slot, so `K == report.partitions.len()`. Reports with mismatched `K` within a group are surfaced as a `warn!` and the group is skipped (planner invariant break). # ballista_core::execution_plans::runtime_stats Two new symbols, exported from `execution_plans::mod`: * `merge_reports(reports) -> Vec<MergedRuntimeStats>`. Groups by `order_by` (prost-encoded bytes as HashMap key), merges T-Digests within each group, computes `K-1` cuts at quantiles `i/K`, returns `MergedRuntimeStats { order_by_len, k, task_count, total_rows, cuts, min, max }`. Empty input → empty output. Sketches with `count() == 0` are dropped from the merge input rather than corrupting the result. * `log_merged_runtime_stats(job_id, stage_id, reports)`. Calls `merge_reports` and formats each group at `debug!` — `RUST_LOG` can promote when verifying against a live cluster. Splits into two log lines depending on whether any sketches were present (drops `min` / `max` in row-count-only mode). `MergedRuntimeStats` is `pub` so slice-D consumers (an AQE rule that plans stage 2 from the cuts) can call `merge_reports` directly without paying the log-formatting cost. # ballista_scheduler::state::execution_stage `RunningStage` gains a `runtime_stats_reports: Vec<RuntimeStatsReport>` field, initialized empty in `RunningStage::new` and in the retry path `FailedStage::to_running` (a fresh attempt discards the previous attempt's stats — merged-cut logging fires per-attempt). New method `append_runtime_stats_reports(reports)` for the graph to push into on each successful task update. # ballista_scheduler::state::execution_graph In the `Successful` arm of `update_task_status`, destructure `SuccessfulTask` to grab both `partitions` and `runtime_stats` and push the reports onto the running stage. When `is_final_successful` flips true (stage transitions to Successful), call `log_merged_runtime_stats` alongside the existing `print_stage_metrics` hook. Reports are dropped on the floor after logging — slice D propagates them onto `SuccessfulStage` when the AQE consumer needs them at plan-time. # Tests Five new tests in `execution_plans::runtime_stats::merge_tests`: * `merge_reports_combines_disjoint_ranges` — two reports over disjoint value ranges; merged sketch spans the union, K=2 midpoint cut falls between the two ranges, total_rows sums. * `merge_reports_k_of_4_produces_three_quartile_cuts` — uniform [0, 100) sample over K=4; the three cuts land in the expected quartile bands. * `merge_reports_row_count_only_emits_empty_cuts` — reports with only row_counts (no sketches) sum correctly with empty `cuts` and `None` `min` / `max`. * `merge_reports_skips_group_with_mismatched_partition_counts` — mismatched `K` within a group drops the whole group. * `merge_reports_empty_input_is_empty_output` — degenerate case. # Verification * `cargo test -p ballista-core --lib` — 175 pass (was 170). * `cargo test -p ballista-scheduler --lib` — 269 pass, 2 ignored. * `cargo test -p ballista-executor --lib` — 53 pass. * `cargo clippy --all-targets` — clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> review: merge_reports returns Result; drop silent per-group fallbacks Address review feedback on the slice-C merge helper: * merge_reports is fallible now. Two failure modes it used to swallow silently — a group with mismatched partition counts, and a corrupted sketch that won't decode — both surface as errors. The scheduler-side caller (log_merged_runtime_stats) catches at warn! and returns; a slice-D consumer that actually plans off the merged cuts can propagate or fail the stage as it sees fit. If the function's job is to merge, it either merges or errors — no "returns empty because the third group was funky" middle ground. * The group-loop body moved into its own `merge_group(&[&Report]) -> Result<MergedRuntimeStats>` helper. The outer loop is one `?` per group; each group's own failure modes live in one place. * `MergedRuntimeStats.k` → `.partition_count`. Struct is public; slice D will consume it. `k` is a math-notation letter, not a field name. * Local single-letter names retired: `r1`/`r2` → `low_range` / `high_range`, `k`/`vs` inside `sketching_report` → `slot_id` / `slot_values`, `make` closure → `make_report`, closure params `|r|` → `|report|`. Per the max-specificity naming preference in memory. * No array accessors in the merge helper or its tests. The empty- group branch of `merge_group` uses `let [first, rest @ ..] = group` and returns an internal error rather than indexing at `[0]`. Tests unpack `merge_reports(...)?` results with `match cuts.as_slice()` and `only_group(...)` helper that panics with a real message rather than `groups[0]`. New test: `merge_reports_propagates_sketch_decode_errors` — builds a wire-shape-corrupt `QuantileSketchState` (3 scalars instead of 6) and asserts the decode error propagates through `merge_reports`. This was the case slice C's original `warn!` + `continue` was hiding. Existing test renamed `merge_reports_skips_group_with_mismatched_...` → `merge_reports_errors_on_mismatched_partition_counts` to reflect the new contract. Verification: * cargo test -p ballista-core --lib — 176 pass (was 175; +1 test). * cargo clippy --all-targets — clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> fix(docs): drop rustdoc links to items outside the doc'd crate CI's `cargo doc --document-private-items --no-deps --workspace` runs with `-D warnings`, which promotes two rustdoc lint classes to errors: * `rustdoc::private-intra-doc-links` — a public item's doc can't link to a private one. `collect_reports` (and its re-export `collect_runtime_stats_reports`) linked to `super::range_repartition_common::preserves_distribution`, but `range_repartition_common` is `mod`, not `pub mod`. * `rustdoc::broken-intra-doc-links` — a link target must resolve in the doc'd crate. `RunningStage::append_runtime_stats_reports` linked to `log_merged_runtime_stats`, which now lives in ballista-core (moved there in the slice-C refactor because ballista-scheduler doesn't depend on `datafusion_functions_aggregate_common`). Both docstrings switched from intra-doc `[...]` links to plain backtick prose that names the target function fully. The prose still points a reader to the right place; the compiler stops complaining. Verified locally with the exact CI invocation: `RUSTDOCFLAGS='-D warnings' cargo doc --document-private-items \ --no-deps --workspace` — clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> review: reshape as_task_status to take TaskCompletionExtras Fold operator_metrics + runtime_stats into a TaskCompletionExtras struct marked #[non_exhaustive] + Default, so future additions are non-breaking for external callers via ..Default::default(). Drops the too_many_arguments allow on as_task_status. Documents the break in docs/source/upgrading/55.0.0.md since the parameter list still changes shape (execution_times now precedes extras, and operator_metrics moves from a positional param into the struct). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix: AQE path forwards SuccessfulTask.runtime_stats to the scheduler `AdaptiveExecutionGraph::update_task_status` in `aqe/mod.rs:774` unpacked `SuccessfulTask.partitions` from an incoming task-success message but silently dropped the `runtime_stats` field. It also skipped `log_merged_runtime_stats` at final-success. The RSE-follow-up wire path was only plumbed on `StaticExecutionGraph`; any tap running under AQE mode (`ballista.planner.adaptive.enabled=true`) had its sketches discarded at the scheduler. Symptom on the adaptive-range-shuffle path: stage-1 tasks correctly harvested one `RuntimeStatsReport` each (visible in executor logs as `Task N finished with … 1 runtime-stats report(s)`), the wire message carried them, but nothing on the scheduler side ever consumed them — `log_merged_runtime_stats` never fired, so the merged cut points that downstream stages need weren't visible. The fix mirrors the static path (`execution_graph.rs:956`): unpack the `runtime_stats` field, hand it to `running_stage.append_runtime_stats_reports`, and call `log_merged_runtime_stats` once the stage is final-successful. Verified on Q20 SF10 — merged output now appears: ``` merged runtime stats: job=… stage=1 order_by_len=1 partition_count=16 task_count=4 total_rows=9088057 cuts=[125078.52, 249995.47, …, 1874842.86] min=1 max=2000000 ``` Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * key runtime-stats reports by producer task_id Wrap each incoming `RuntimeStatsReport` in a `TaskRuntimeStats { producer_task_id, report }` when the scheduler accumulates it on `RunningStage.runtime_stats_reports`. Threads the producer `task_id` through both accumulation paths (static- and adaptive-graph `update_task_status`). Motivation: reports are appended on every successful-task update, but a task's "success" can be invalidated later (executor loss → task marked `Failed(ResultLost)`, partition rescheduled). Without a producer key on the accumulated entries, the merged view at stage-final-success would double-count the reset producer's contribution — the old ghost plus the retry's fresh report. Keying by producer `task_id` gives the follow-up purge helper something to filter on. No behavior change here (purge lands in the next commit). `log_merged_runtime_stats` extracts the raw reports internally for the existing merge, so the logged output is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * purge stale runtime-stats reports when a task is reset for retry `RunningStage.runtime_stats_reports` accumulates per-task sketches on every successful-task update, but a task's "success" is not permanent — an executor loss flips previously-successful tasks to `Failed(ResultLost)` and reschedules their partitions on another executor. Without a purge, the ghost reports stayed in the vec; the retry appended its fresh reports under a new `task_id`; the merged view at stage-final-success double-counted the bounced slices. Addresses reviewer feedback on this PR (Phillip LeBlanc, apache#2175): "Reports are not associated with task attempts. When successful work is invalidated and rerun (i.e. lost executor, and the scheduler requeues the task), the old report cannot be removed." They are now — each accumulated entry carries the producer `task_id`, so both reset paths on `RunningStage` filter out stale contributions: - `reset_task_info(task_id)`: retryable single-task reset (task killed / result lost). Drops entries where `producer_task_id == task_id`. - `RunningStage::reset_tasks(executor)`: whole-executor loss on a running stage. Collects the set of reset task_ids while flipping their statuses, then filters the reports vec against that set in one pass. `SuccessfulStage::reset_tasks` doesn't need matching treatment — `SuccessfulStage` doesn't carry the reports (they were consumed by `log_merged_runtime_stats` at stage-final-success), and `to_running` starts the next attempt with a fresh empty vec. Two unit tests cover both paths: single-task reset purges only the matching entry; executor-loss reset purges every entry from the lost executor while leaving surviving executors' entries alone. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
…ts (apache#2186) * fix: stop every task running every union branch at low partition counts Per-task restriction gives each branch of a `UnionExec` the partitions that task owns and an empty slice to the rest, relying on an unowned branch becoming 0-partition so the union's index arithmetic routes past it. That fails when a branch's partition count does not come from restrictable leaves. A `CoalescePartitionsExec` or a non-preserving `SortExec` reports one partition whatever its leaves do, and a broadcast `ShuffleReaderExec` is deliberately never pruned. The branch keeps reporting a partition and keeps producing all of its data, so the executor -- which runs every partition of the plan it is handed -- runs every branch for every task. The result is a silently inflated answer, or a task executing a branch whose scan was emptied. At SF1 with `target_partitions=1`, TPC-DS q5 came back 4x and q80 2x, and q49 failed with `FileStreamBuilder invalid partition index`. It does not show at higher partition counts because those union stages are capped by a hash repartition, where the writer's output is an intrinsic K-space and inner partition counts do not matter. Cut a stage boundary beneath a union branch that cannot be restricted away. The branch becomes a `ShuffleReaderExec`, which restricts to nothing cleanly, and the collapse moves into its own stage where it still sees its whole input. Branches that already restrict away stay inline, so a union of plain scans is unaffected. Whether a branch can be restricted away is decided by asking the rewriter rather than by matching operator types, so it stays correct as the rewriter evolves. Branches holding a broadcast input or a `CollectLeft` join are treated as unrestrictable regardless: those build and probe sides can still be swapped after stage planning, and the two orders restrict differently, so a branch that looks restrictable at planning time may not be by the time the task is built. Also take restriction scope from `required_input_distribution()` for every operator instead of a list of join types. `CrossJoinExec` collects its left input but was not on that list, so each task of a multi-partition stage rebuilt the cross join's left side from its own slice. That surfaced as an inflated q77 (`from cs, cr`) once its branch got a stage of its own. Verified at SF1 against single-process DataFusion: the full suite passes at `--partitions 16`, and at `--partitions 1` only q72 remains, exhausting its per-task memory share rather than returning a wrong answer. The adaptive planner builds stages through its own path and is not covered by this change; it still fails these queries at one partition. * refactor(scheduler): tidy the union stage-boundary fix after review No behaviour change; verified by re-running TPC-DS SF1 at both partition counts (full suite passes at 16, only the pre-existing q72 memory failure at 1). - Reuse the generic child-recursion loop instead of repeating it in the UnionExec branch. The union case now rewrites the collected children in place, so the recursion contract lives in one spot. - Rename the branch predicate to `can_stay_inline`. It answered "can this be restricted away?" in its name while also folding in the broadcast guard, so the name promised less than the function did. `restricts_to_nothing` is now just the rewriter query it claims to be. - Share `scan_with_file_groups` from `test_utils` rather than keeping a near-identical copy in each of the two test modules. - Make the broadcast test exercise the guard. It asserted on a bare broadcast node, which the rewriter already reports as unrestrictable, so it passed with the guard deleted. It now checks detection through a nested branch, which is where the guard actually earns its keep. - Record why the `CoalescePartitionsExec` / `SortPreservingMergeExec` case cannot be folded into the rule below it: both declare `UnspecifiedDistribution`, so the general path would miss them.
Standalone operator carrying `Vec<Arc<dyn PhysicalExpr>>` — one boolean predicate per input partition — applied on `execute(k)`. Preserves vcore packing over the one-task-per-partition alternative, which would explode task count to K and defeat the range-repartition family. Not yet wired into any planner/rule; that lands in a follow-up slice alongside cuts-on-ExchangeExec and injection at the URRE-consuming stage. PerPartitionFilterExec: bounds-check partition and eager-release input on EOS Two small defensive tweaks to match DataFusion's FilterExec pattern: - `execute(partition)` now returns `internal_err!` if `partition` is out of range instead of panicking on the Vec index. The invariant is `partition < output_partitioning().partition_count()`, which construction guarantees against the predicate count — but ExecutionPlan callers indexing wrongly should get an error, not a panic. Uses `let-else` on `predicates.get(...).cloned()` so the guard and the clone happen in one step. - On end-of-stream in `poll_next`, replace `self.input` with an `EmptyRecordBatchStream` before returning `Poll::Ready(None)`. Mirrors DF's FilterExec (see `filter.rs:1027`) so the input's operator chain doesn't stay alive on the heap while the outer stream is still held. Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Null-aware LeftAnti joins cannot be swapped and DataFusion coordinates their visited-row and NULL state only within one process. Preserve CollectLeft, collect the build side, and coalesce the probe side so static and AQE plans run the join in exactly one task. Reject disabled or known oversized single-task builds through the Ballista broadcast threshold. Add optimizer, stage-lowering, threshold, and standalone distributed NOT IN regression tests. Closes apache#2187 Signed-off-by: Phillip LeBlanc <[email protected]>
…embedded projections, and dropped exchange partitioning (apache#2200) * test(scheduler): show PropagateEmptyExecRule collapses GROUPING SETS(()) DataFusion's logical PropagateEmptyRelation guards on both `!agg.group_expr.is_empty()` AND `!has_empty_grouping_set(&agg.group_expr)`. The physical port here only carries the first guard, so a GROUPING SETS / ROLLUP / CUBE containing the empty subset `()` still collapses to EmptyExec over an empty input — losing the all-NULL row(s) the empty subset would emit. Adds a failing test that constructs `GROUPING SETS ((a), ())` over an EmptyExec and expects the aggregate to be preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(scheduler): harden PropagateEmptyExecRule against grouping sets, embedded projections, and dropped exchange partitioning Preserve aggregates whose grouping contains the empty grouping set () from GROUPING SETS/ROLLUP/CUBE, mirroring has_empty_grouping_set in DataFusion's logical PropagateEmptyRelation rule. On DataFusion 54 the empty subset emits one grand-total row even over zero input rows, so collapsing the aggregate to EmptyExec drops that row. Skip the join rewrites that keep one input alive (null-padded projection for Left/Right/Full, raw-child replacement for anti joins) when the hash join carries an embedded projection: the join schema then no longer maps positionally onto the input schemas, so null-padding reads the wrong columns and child replacement changes the schema. Arms that produce an EmptyExec use the join's own projected schema and remain unguarded. Replace a FilterExec over an empty input with an EmptyExec using the filter's schema instead of returning the input, since a filter with an embedded projection has a different schema than its input. Replace an ExchangeExec over an empty input with an EmptyExec that keeps the exchange's output partition count instead of returning the input, matching the stats-based exchange arm below it. --------- Co-authored-by: Brent Gardner <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
…ans (apache#2199) * feat(core): rewrite NOT IN subqueries into distributable anti join plans Add a NotInSubqueryRewrite logical optimizer rule that rewrites uncorrelated NOT IN (subquery) filter conjuncts into a plain LeftAnti join plus a one-row count aggregate before DataFusion's subquery decorrelation can plan a null-aware hash join. The null-aware join coordinates probe-side state through in-process atomics and therefore forces single-task execution in Ballista; the rewritten plan uses only ordinary joins and aggregates and distributes normally. The rule runs first in the logical optimizer chain and is registered in every Ballista session builder (client state creation and upgrade, and the scheduler's default session builder). It is enabled by default and controlled by ballista.optimizer.not_in_subquery_rewrite. The rewrite only fires where null-aware semantics would be needed: uncorrelated single-column subqueries with a nullable key on either side. Correlated, multi-column, volatile, and provably non-nullable predicates are left to DataFusion's own decorrelation. This also fixes wrong NOT IN results under Ballista's default sort-merge-join configuration, where the null-aware flag was previously dropped before scheduler lowering. * fix(core): use explicit path for intra-doc link in optimizer module docs The outer doc comment on 'pub mod optimizer' in lib.rs is merged with the module's inner docs, and rustdoc resolves the merged links in the parent scope where the bare item name is not visible.
…e paths on a real cluster (apache#2026)
…2203) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.85.2 to 2.85.3. - [Release notes](https://github.com/taiki-e/install-action/releases) - [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md) - [Commits](taiki-e/install-action@41049aa...18b1216) --- updated-dependencies: - dependency-name: taiki-e/install-action dependency-version: 2.85.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.42 to 0.23.43. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](rustls/rustls@v/0.23.42...v/0.23.43) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.43 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [http](https://github.com/hyperium/http) from 1.4.2 to 1.5.0. - [Release notes](https://github.com/hyperium/http/releases) - [Changelog](https://github.com/hyperium/http/blob/master/CHANGELOG.md) - [Commits](hyperium/http@v1.4.2...v1.5.0) --- updated-dependencies: - dependency-name: http dependency-version: 1.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ctor](https://github.com/mmastrac/linktime) from 1.0.11 to 1.0.12. - [Release notes](https://github.com/mmastrac/linktime/releases) - [Changelog](https://github.com/mmastrac/linktime/blob/master/CHANGELOG.md) - [Commits](mmastrac/linktime@ctor-1.0.11...ctor-1.0.12) --- updated-dependencies: - dependency-name: ctor dependency-version: 1.0.12 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(cli): sort jobs by stage completion ratio Signed-off-by: QuakeWang <[email protected]> * fix(cli): simplify stage completion sorting Signed-off-by: QuakeWang <[email protected]> --------- Signed-off-by: QuakeWang <[email protected]>
…2205) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.85.3 to 2.85.4. - [Release notes](https://github.com/taiki-e/install-action/releases) - [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md) - [Commits](taiki-e/install-action@18b1216...065d6a0) --- updated-dependencies: - dependency-name: taiki-e/install-action dependency-version: 2.85.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [nix](https://github.com/nix-rust/nix) from 0.29.0 to 0.31.3. - [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md) - [Commits](nix-rust/nix@v0.29.0...v0.31.3) --- updated-dependencies: - dependency-name: nix dependency-version: 0.31.3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.28 to 0.13.4. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](seanmonstar/reqwest@v0.12.28...v0.13.4) --- updated-dependencies: - dependency-name: reqwest dependency-version: 0.13.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… does (#72) try_update, its replacement, is newer than the workspace MSRV (1.94).
sgrebnov
approved these changes
Oct 2, 2026
|
Addressed #68 (review). No new commit: HEAD stays That review had one line thread (python/Cargo.toml, answered at #68 (comment)). Its headline was that the Python dependencies remain on DataFusion 54, health-server failures are ignored, and benchmark and URL-handling regressions remain. Every file it names is upstream's, merged verbatim from apache/datafusion-ballista@da205206. For each one,
The health-server, URL-encoding and Python-keyword points weren't evaluated as defects here (unverified, code inspection only: Copilot's reading). If they're real, the fix belongs upstream, and a later sync would bring it in. License compliance: 3 open alerts need a license-policy decision, not a commit. The dependency-review diff between base
CodeQL: the 18 open Addressed #68 (comment) (the RAT CI note) by checking it on this HEAD: |
* add_col_stats_framework * add_col_stats_framework * add_col_stats_framework * add_col_stats_framework * consume_col_stats_successful_stage * consume_col_stats_successful_stage * add_tests * add_tests * fix_lint * fix_lint * fix_lint * fix_lint * propogate_column_stats_along * address_review_comments * make stricter fallbacks and add further tests lint * make col stats encapsulated * address review comments and add default col stats method across carriers * address review feedback
…ns (apache#2514) * feat: error when the client and scheduler have different major versions A client and a scheduler from different major versions can run a query and return wrong results, because DataFusion's plan encoding can change between majors in ways that still decode. Until now a mismatch only logged a warning. The client now sends its version in a `ballista-version` gRPC header with `ExecuteQuery` and `ExecuteQueryPush`. The scheduler rejects a different major version with `failed_precondition` before it decodes the plan, and sends its own version back. A side that sends no header is treated as older than 55.0.0, so 54.x clients are rejected once the scheduler is on 55.0.0. The client also checks the scheduler's reply. A scheduler older than 55.0.0 doesn't check versions and has already queued the job, so the client cancels it before returning the error. Closes apache#2513. * docs: state that clients and schedulers stay compatible within a major version The version check treats the major version as the unit of compatibility, so a change that would break an older client of the same major version has to wait for the next major release. Say so next to the check, next to `BALLISTA_PROTOCOL_VERSION` (which can change in any release), and in the upgrade guide. * fix: remove redundant rustdoc link targets
…che#2416) * feat: experimental Arrow Flight SQL frontend (WIP) Adds a new `ballista-flight-sql` crate implementing an Arrow Flight SQL frontend, and wires it into the scheduler behind a non-default `flight-sql` feature and a `--flight-sql` runtime flag. This lets generic SQL clients (Python ADBC, JDBC/ODBC, BI tools) run distributed queries without linking against Ballista. Flight SQL was removed in 46.0.0 (apache#1227, apache#1228) because it was unmaintained, incomplete, coupled to `SchedulerServer`, and untested. This is a fresh implementation rather than a revival: the frontend talks to a `QueryBackend` trait and never sees scheduler internals, endpoints carry no location so no executor address reaches a client, and catalog metadata comes from the session's real DataFusion catalog. `flight_proxy_service.rs` moves from `ballista/scheduler/src` to `ballista/core/src` unchanged, so both crates can use it. Part of apache#2298. * refactor(flight-sql): cache sessions, unify plan classification, drop dead API Cleanup pass over the Flight SQL frontend. The important one is session caching. `QueryBackend::session` is not a lookup — `InMemoryJobState::create_or_update_session` builds a fresh `SessionState` and `RuntimeEnv` on every call — so the frontend was constructing a whole DataFusion session per RPC, and a table registered by one request was invisible to the next. `SessionStore` now caches the context per session id under the same TTL and reaper as the other handles. Also: - Replace the `runs_on_scheduler` / `unsupported_reason` predicate pair with one `disposition()` classifier, so the "check unsupported first" ordering that CTAS depends on cannot be got wrong at a call site. - Delete `with_session_ttl` and `with_advertised_endpoint`: neither had a caller, and `with_session_ttl` orphaned the store and reaper task that `new` had just created. - Give the authenticator a real seam. `SchedulerConfig::with_flight_sql` and `with_flight_sql_authenticator` make the documented extension point reachable without reimplementing `start_grpc_service`, and `flight_sql_enabled()` moves the `cfg` pair next to the field it reads. `validate()` now reports the frontend subsuming the embedded proxy. - Share the shuffle-layout rule via `protobuf::PartitionLocation::layout()` in ballista-core instead of restating it, and fold `metadata::one`, `session_context`, and the repeated ticket/session error mapping into single helpers. - Drop the per-eviction reference scan in `SessionStore::remove_session`: each token gets its own session, so it could never fire, and it made `sweep` quadratic. - Test fixture `client_with_people` replaces the preamble copied across three tests, and a boxed error type removes ~17 `map_err` wrappers. * fix(ci): satisfy prettier, and add a TPC-H runner over Flight SQL Prettier failed on the markdown this branch adds. Two of the fixes were its own normalisation (link reference labels), but it also had to join lines I had wrapped in the middle of inline code spans; those bullets are reworded so its output stays readable. `benchmarks/tpch_adbc.py` runs the 22 TPC-H queries through the Flight SQL frontend with the generic ADBC driver and no Ballista client library, so it measures the path a JDBC/ODBC or BI-tool client takes. It mirrors the Rust harness where the two must agree: the same table list, the same `find_path` resolution for object-store URLs, the same statement splitting, and the same "last SELECT or WITH is the answer" rule that q15 needs. Timings are reported separately for GetFlightInfo and DoGet, because GetFlightInfo blocks for the whole query until PollFlightInfo is implemented, which is what a `--timeout` has to be sized against. * fix(flight-sql): advertise the schema the data actually carries `GetFlightInfo` described the result with the logical plan's schema while `DoGet` delivered the physical one. The two disagree about nullability often enough to matter: `version()` is nullable logically and not-null in the executed batches, and every list, struct and map literal goes the other way. A generic client that is promised one schema and handed another rejects the result outright, so `SELECT [1,2,3]` and friends failed against the ADBC Flight SQL driver. Take the schema from the physical plan instead. That leaves `EXPLAIN`, which the scheduler answers with a plan of its own whose two columns were nullable while both DataFusion's `ExplainExec` and `DistributedExplainAnalyzeExec` declare them not-null; make it agree, so `EXPLAIN` describes itself the same way however it runs. Every element it unnests is one of the three plan strings, so none can be null. * fix(flight-sql): answer information_schema queries on the scheduler `SHOW TABLES`, `SHOW ALL` and any other `information_schema` read was sent to the cluster, where its physical form is a `StreamingTableExec` the codec cannot serialize. The tasks were never launched, no status ever came back, and `GetFlightInfo` hung forever -- which is what a BI tool does on connect. `BallistaQueryPlanner` already solves this with its `LocalRun` visitor, which is why `ballista-cli` handles `SHOW TABLES` fine. Expose that as `scans_only_local_tables` and have the frontend's `disposition` use it, so there is one definition of a plan with nothing to distribute. * fix(scheduler): fail jobs whose tasks cannot be launched `launch_multi_task` logged a task-preparation failure and dropped the tasks. Nothing was sent to an executor, so no status ever came back and the job sat Running for the life of the scheduler, with every caller awaiting it -- a Flight SQL `GetFlightInfo`, a `DistributedQueryExec` -- blocked with no timeout. Report the jobs whose definitions could not be prepared and fail them, which is what the pull-based `poll_work` path already does. Keep that distinct from `launch_tasks` failing, which says the executor is sick rather than the plan. * fix(benchmarks): make the ADBC scripts re-runnable Unauthenticated clients share one session, so a scheduler that has already served the benchmark or the example still has the tables registered and the second run died on "External table 'part' already exists". Drop each table before creating it. * fix(flight-sql): address review feedback - Remove CancelQuery: a client only holds the FlightInfo once the job has finished, and cancelling then deleted its results. Report FlightSqlServerCancel as false. - Report SqlIdentifierCase / SqlQuotedIdentifierCase with the named enum values (LOWERCASE / CASE_INSENSITIVE) instead of swapped literals. - Refuse Dml, Copy, CTAS and EXECUTE anywhere in the plan, subqueries included, so EXECUTE and EXPLAIN ANALYZE INSERT no longer run on the scheduler. - Walk subqueries in scans_only_local_tables. - Check the bearer token in every Flight SQL handler and bind prepared statements and local results to their session. Document that the authenticator isolates sessions rather than securing the port. - Take the result schema from the job's final stage instead of planning the query a second time. - Build sessions lazily and start the reaper on first request, so the service can be constructed outside a Tokio runtime. - Sweep sessions in a single retain pass and backdate timestamps in the TTL test instead of sleeping. - Map undecodable tickets to InvalidArgument in the proxy and drop the duplicate decode in do_get_fallback. Reuse PartitionLocation::layout in fetch_partition. - Consolidate and de-duplicate tests; add coverage for multiple endpoints, auth on every handler, cross-session isolation through a real handshake, and the refused statements. - Run the Flight SQL end-to-end tests in the main CI test step. * chore: move the ADBC example and TPC-H runner to a follow-up PR `examples/python/adbc_flight_sql.py` and `benchmarks/tpch_adbc.py`, along with their README sections and the user guide's links to them, move to a PR stacked on this one so this PR stays focused on the frontend. The user guide keeps its inline ADBC walkthrough. * docs(flight-sql): describe the result schema and port security accurately `QueryResult::schema` said it was taken from the submitted plan. It is the schema of the data in the result partitions, which the scheduler reads from the job's final stage because the logical plan's can disagree with the data on nullability. The `--flight-sql` help implied that an authenticator secures the port. It only isolates sessions: the port also serves unauthenticated gRPC, REST and partition fetches. Say to keep it on a trusted network, as the user guide and crate README already do.
|
There is a v55 release tag now. |
Brings in the 13 upstream commits between da20520 and the 55.0.0-rc1 tag (Flight SQL frontend + ADBC, client/scheduler major-version check, column statistics v1, a setting to disable the shared file statistics cache, the 55.0.0 version bump and release tooling, Rust 1.99 clippy fixes, CI and Python dependency bumps). Conflicts, kept both sides: - core distributed_query.rs: import ResultFetchMetricsCallback (fork) and BallistaGrpcMetadataInterceptor (upstream apache#2514). - executor executor.rs: upstream's ShuffleWritePartition import, fork's task metrics recording. - scheduler grpc.rs: fork's grouped scheduler import plus upstream's version imports. - scheduler execution_stage.rs: SuccessfulStage keeps the fork's encoded_plan and gains upstream's output_column_stats. Integration fixes the merge needed to compile: - SuccessfulStage::decode (fork-only persistence) rebuilds output_column_stats from the decoded task statuses through fold_output_column_stats, now shared with to_successful; a round-trip test covers it. - ballista-flight-sql passes the partition location's path (fork field) into FetchPartition, so memory:// and object-store partitions resolve; its test asserts it. - Upstream test code: stub the fork's get_catalog/get_remote_functions RPCs in the version-check mock, and set the fork's TaskInfo.executor_id in the column-stats test helper.
7 tasks done
Merge upstream Ballista 55.0.0-rc1
lukekim
added a commit
to spiceai/spiceai
that referenced
this pull request
Oct 3, 2026
spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. #68 is still open, waiting on license-compliance decisions. Pin 718429f, the #73 merge, until #68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1.
lukekim
disabled auto-merge
October 3, 2026 20:56
lukekim
added a commit
to spiceai/spiceai
that referenced
this pull request
Oct 3, 2026
spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 718429f, the previous pin (`git diff 718429f a7c4c58` is empty). Repoint the pin at the landed revision and drop the TEMPORARY marker from the fork-patch ledger, as scripts/check_fork_patches.py asks. The ledger's ballista section now names spiceai-55 and upstream 55.0.0-rc1 as its base. Cargo.lock changes only the source revision of the five ballista packages.
krinart
added a commit
to spiceai/spiceai
that referenced
this pull request
Oct 3, 2026
anush008
pushed a commit
to anush008/spiceai
that referenced
this pull request
Oct 6, 2026
* build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing spiceai#237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for spiceai#237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (spiceai#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (spiceai#70, spiceai#71, spiceai#72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (spiceai#82, spiceai#53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, spiceai#54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, spiceai#15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, spiceai#16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, spiceai#7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, spiceai#50), datafusion-table-providers 465926a3 (spiceai-55, spiceai#80), snowflake-rs f5557381 (spiceai-59, spiceai#13), spark-connect-rs 18ae9bd3 (spiceai-59-2, spiceai#15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (spiceai#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: spiceai#49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. spiceai#68 is still open, waiting on license-compliance decisions. Pin 718429f, the spiceai#73 merge, until spiceai#68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, spiceai#73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of spiceai#68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 718429f, the previous pin (`git diff 718429f a7c4c58` is empty). Repoint the pin at the landed revision and drop the TEMPORARY marker from the fork-patch ledger, as scripts/check_fork_patches.py asks. The ledger's ballista section now names spiceai-55 and upstream 55.0.0-rc1 as its base. Cargo.lock changes only the source revision of the five ballista packages. * perf(cayenne): keep small Vortex scans whole on the main query path DataFusion 55 lowered `repartition_file_min_size` from 10 MiB to 1 MiB (apache/datafusion#22439), so the main Cayenne scan byte-range-splits a 1-10 MiB dimension table `target_partitions` ways and pays a Vortex footer open per range. The main scan now opts small scans out below 10 MiB, as the internal and protected-snapshot scans already do below 256 MiB; Parquet listing scans keep the new default. `small_snapshot_groups_opt_out_of_repartitioning` now asserts the main scan's opt-out and fails with the threshold back at 0. * docs(fork_patches): record duckdb-rs on the protected spiceai-1.4.4-patches-2 head, which table-providers pins * test(duckdb): build CreateExternalTable with DataFusion 55's locations field * ci: validate the datafusion-table-providers pin against spiceai-55, where spiceai#80 landed * test(cayenne): size the point-lookup fan-out control above the main scan's 10 MiB opt-out * fix(data-accelerator-api): implement DataFusion 55's apply_expressions for KeepFirstExec * test: port trunk's new tests to DataFusion 55 (CreateExternalTable locations, common::TableReference) * fix(cayenne): keep -0.0 and 0.0 equal in pushed-down float comparisons and memory-mode indexes DataFusion 55 compares floats with the two zeros equal (apache/datafusion#22835), while Vortex and the Arrow row encoding order them apart. Spell out both zeros when a float comparison or IN list with a zero literal is pushed into a Vortex scan, keep float column-vs-column comparisons above the scan, and hash memory-mode index keys with the file-mode KeyEncoder. * Fix lint --------- Co-authored-by: Luke Kim <[email protected]>
peasee
pushed a commit
to spiceai/spiceai
that referenced
this pull request
Oct 8, 2026
…n fixes backported from 54 (#14800) * build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing #237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for #237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (#70, #71, #72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (#82, #53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, #54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, #15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, #16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, #7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, #50), datafusion-table-providers 465926a3 (spiceai-55, #80), snowflake-rs f5557381 (spiceai-59, #13), spark-connect-rs 18ae9bd3 (spiceai-59-2, #15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: #49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. #68 is still open, waiting on license-compliance decisions. Pin 718429f, the #73 merge, until #68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, #73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of #68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 718429f, the previous pin (`git diff 718429f a7c4c58` is empty). Repoint the pin at the landed revision and drop the TEMPORARY marker from the fork-patch ledger, as scripts/check_fork_patches.py asks. The ledger's ballista section now names spiceai-55 and upstream 55.0.0-rc1 as its base. Cargo.lock changes only the source revision of the five ballista packages. * perf(cayenne): keep small Vortex scans whole on the main query path DataFusion 55 lowered `repartition_file_min_size` from 10 MiB to 1 MiB (apache/datafusion#22439), so the main Cayenne scan byte-range-splits a 1-10 MiB dimension table `target_partitions` ways and pays a Vortex footer open per range. The main scan now opts small scans out below 10 MiB, as the internal and protected-snapshot scans already do below 256 MiB; Parquet listing scans keep the new default. `small_snapshot_groups_opt_out_of_repartitioning` now asserts the main scan's opt-out and fails with the threshold back at 0. * docs(fork_patches): record duckdb-rs on the protected spiceai-1.4.4-patches-2 head, which table-providers pins * test(duckdb): build CreateExternalTable with DataFusion 55's locations field * ci: validate the datafusion-table-providers pin against spiceai-55, where #80 landed * test(cayenne): size the point-lookup fan-out control above the main scan's 10 MiB opt-out * fix(data-accelerator-api): implement DataFusion 55's apply_expressions for KeepFirstExec * test: port trunk's new tests to DataFusion 55 (CreateExternalTable locations, common::TableReference) * fix(cayenne): keep -0.0 and 0.0 equal in pushed-down float comparisons and memory-mode indexes DataFusion 55 compares floats with the two zeros equal (apache/datafusion#22835), while Vortex and the Arrow row encoding order them apart. Spell out both zeros when a float comparison or IN list with a zero literal is pushed into a Vortex scan, keep float column-vs-column comparisons above the scan, and hash memory-mode index keys with the file-mode KeyEncoder. * build: move to iceberg-rust 0.11.0 (RC) and opendal 0.58 Points the iceberg crates at spiceai/iceberg-rust#55, which merges the upstream 0.11.x release branch onto the DataFusion 55 fork line. opendal moves to 0.58 to match iceberg-storage-opendal, so one opendal stack is built instead of two. 0.58 removes the OperatorBuilder step: Operator::new returns the operator with the same default layers that finish() applied. * Bump datafusion to 55.2-rc1 * build: pin iceberg-rust at spiceai-0.11.0-df-55 (6317d03e) spiceai/iceberg-rust#55 merged into spiceai-0.11.0-df-55. Same Rust code as the previous pin (e02547cb); the branch adds only a Python CI pin and a doc-comment fix. fork_patches.md records the new branch and that SigV4 is now an AuthManager in the fork's REST catalog. * build: keep the lockfile's unrelated resolutions from the DataFusion bump The previous commit's lockfile update also re-resolved five crates with range requirements (hashbrown, heck, base64). Restore them so the iceberg repin changes only the iceberg-rust source lines. * build: pin iceberg-rust at spiceai-0.11.0-df-55 (7735caa2) Picks up spiceai/iceberg-rust#49 (IcebergTableProvider::catalog and ::table_ident) and #56 (the fork's own DataFusion pin, which does not affect this build). #49's fork_patches row and its compile guard land with #14588, which first calls the getters. * build: pin datafusion at spiceai/datafusion#250 (55.2.0-rc1 and the 54 backports) Moves the DataFusion fork from 02550cf9 (55.1.0) to 1749ce4e, the head of spiceai/datafusion#250, which merges into #249: `spiceai-55` at 55.2.0-rc1 (spiceai/datafusion#248) plus the `spiceai-54` patches #233, #234 and #245 carried onto 55, and #250's review fix. The guards for those follow in the next commit. The pin names a pull request branch, so the ledger row is marked TEMPORARY and `scripts/check_fork_patches.py` refuses to let it land until #250 and #249 merge and the pin moves to the `spiceai-55` merge commit. The 55.2.0-rc1 merge changes no Spice patch. Outside `Cargo.lock`, the fork's 02550cf9..27351ab0 diff has the same patch-id as upstream's 55.1.0..55.2.0-rc1 diff, except `datafusion-spark`'s `quote.rs`, which the fork's backport had already made identical to 55.2.0-rc1's. That backport's ledger row is dropped; upstream carries the fix as apache/datafusion#25277. `Cargo.lock` changes only the 36 DataFusion entries. * test: guard the DataFusion unparser and footer-cache fixes backported from 54 Each backported fix gets a guard here, so the next re-cut of the fork cannot drop it silently, plus a ledger row naming the guard. - spiceai/datafusion#233 (#14373): a join that is another join's right input stays a parenthesised joined table on that join's right. `a_join_that_is_another_joins_right_input_stays_on_its_right` checks that `b` is introduced before the outer `ON` names it and that a LEFT JOIN keeps a filter from inside it out of `WHERE`. With the `sqlite` feature it runs the SQL on SQLite and compares the rows with DataFusion's for the same plan. - spiceai/datafusion#234 (#14375): a `Limit` that is a join input bounds that input, not the join. `a_limit_on_a_join_input_bounds_that_input_rather_than_the_join`, with the same row comparison. - spiceai/datafusion#250: a projection over a join that passes up a mark join's mark is refused rather than unparsed with the mark unbound. `a_mark_a_projection_over_a_join_passes_up_is_refused`. - spiceai/datafusion#245 (#12952): the file metadata cache frees an evicted footer's hit counter. 55's `DefaultCache` already does, so 55 carries no patch; `crates/cayenne/tests/footer_cache_hit_counter_test.rs` pins the behaviour by counting the live heap across 20,000 cold footer reads into a full cache. At 02550cf9 the three unparser guards fail. SQLite rejects the nested join's SQL ("ON clause references tables to its right"), returns 1 row where the plan returns 2 for `b FULL JOIN (c LIMIT 1)`, and the mark join renders as `… LEFT OUTER JOIN (b) ON a.id = b.id WHERE NOT c.mark`. Against a local DataFusion build with the hit-counter prune removed, the footer-cache guard measures 2,646,176 bytes of heap growth. At 1749ce4e all four pass, and the footer-cache growth is 0 bytes. * build: pin datafusion at spiceai/datafusion#249's merge commit Moves the fork pin from #250's branch head to bebc4d58, the merge of #249 into `spiceai-55`. Its tree is #249's head, cbda233a: 55.2.0-rc1 plus the backported #233, #234 and #245. The pin names a landed revision, so the TEMPORARY marker comes off and `scripts/check_fork_patches.py` passes. #250 is not part of this revision, so its guard and ledger row are removed. * Update datafusion version to 55.2 in Cargo.toml * docs(fork_patches): record the unparser frame refactor #249 carries `cbda233a6` is the fourth Spice commit at the pinned revision. It moves the join-input bookkeeping that #233 and #234 add out of `select_to_sql_recursively_inner`, so an unoptimised build's frame stays where it was. It had no ledger row, so a re-cut could drop it without the audit noticing. Its row is a GAP, with an Open gaps entry. What it buys is stack headroom in a debug build: without it, the fork's `roundtrip_statement` needs 2,112 KiB and overflows a 2 MiB test thread. This repo runs tests on 8 MiB threads, so no test here sees the difference deterministically. The entry names the fork-side check to run at a re-cut instead. * Fix tests * build: pin DataFusion at spiceai/datafusion#251, so MAX of a partition column skips empty partitions (#14801) * build: pin datafusion at spiceai/datafusion#251's merge commit Moves the DataFusion fork pin from #249's merge commit to the head of `spiceai-55`, which adds two patches on top of it: - #251: a file that holds no rows gets no partition-column min/max. Without it, `MIN`/`MAX` of a partition column is answered from the listing's statistics with the value of a partition whose files are all empty: `max(p)` answers '99' for a `p=99` directory holding one empty file, where the rows give '3'. - #240: the listing prunes files by metadata-column predicates before opening them and reports those filters `Exact` (#229 on `spiceai-54`). Neither changes a manifest, so `Cargo.lock` only moves its source revision. Each patch gets a ledger row and a guard in the listing connector's tests, which the gate runs as library tests. * fix(listing): keep every predicate residual on the `_location` fast path The pin now carries spiceai/datafusion#240, so the inner listing reports a metadata-column predicate such as `_size < 50` `Exact`. `MetadataPruningListingTable` handed pushdown classification to the inner listing whenever a `_location` predicate was present, but its `_location` fast path opens the named objects and applies no other predicate, so `_location = x AND _size < 50` returned the rows of a file the `_size` predicate excludes. Report every predicate `Inexact` on that path as well, so a residual `FilterExec` re-applies it. This is #14790's change, which makes the same fix for partition predicates, taken byte for byte so that whichever of the two lands second merges cleanly. The new test goes through `create_listing_table` with `_location` and `_size` enabled. * test(metadata): expect the listing to answer `_size = 2319` exactly With spiceai/datafusion#240 the listing prunes files by a metadata-column predicate and reports it `Exact`, so the plan for `met_size`, which enables only `_size`, has no residual `Filter`. The scan lists only the four 2319-byte files; `data_0.parquet` is 2317 bytes, as its S3 `Content-Length` shows. Regenerated by running `metadata::s3_metadata_columns` and accepted with `cargo insta accept`. The rest of the test passes unchanged at this pin, including the `_location` plans and the returned rows. * fix(data-connector-api): satisfy pedantic clippy in the metadata-pruning listing test `make lint-rust`'s `--tests` pass rejected two lints in `listing_table_prunes_files_by_a_metadata_column_predicate`: `redundant_closure_for_method_calls` (`|group| group.len()`) and `format_collect` (`map(format!).collect::<String>()`). Use `FileGroup::len` and build the file contents with `writeln!` into one `String`. The test is unchanged in what it checks. --------- Co-authored-by: Luke Kim <[email protected]> Co-authored-by: Sergei Grebnov <[email protected]>
wangyusheng1985
pushed a commit
to wangyusheng1985/spiceai
that referenced
this pull request
Oct 8, 2026
…n fixes backported from 54 (spiceai#14800) * build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing spiceai#237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for spiceai#237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (spiceai#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (spiceai#70, spiceai#71, spiceai#72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (spiceai#82, spiceai#53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, spiceai#54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, spiceai#15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, spiceai#16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, spiceai#7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, spiceai#50), datafusion-table-providers 465926a3 (spiceai-55, spiceai#80), snowflake-rs f5557381 (spiceai-59, spiceai#13), spark-connect-rs 18ae9bd3 (spiceai-59-2, spiceai#15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (spiceai#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: spiceai#49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. spiceai#68 is still open, waiting on license-compliance decisions. Pin 718429f, the spiceai#73 merge, until spiceai#68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, spiceai#73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of spiceai#68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 718429f, the previous pin (`git diff 718429f a7c4c58` is empty). Repoint the pin at the landed revision and drop the TEMPORARY marker from the fork-patch ledger, as scripts/check_fork_patches.py asks. The ledger's ballista section now names spiceai-55 and upstream 55.0.0-rc1 as its base. Cargo.lock changes only the source revision of the five ballista packages. * perf(cayenne): keep small Vortex scans whole on the main query path DataFusion 55 lowered `repartition_file_min_size` from 10 MiB to 1 MiB (apache/datafusion#22439), so the main Cayenne scan byte-range-splits a 1-10 MiB dimension table `target_partitions` ways and pays a Vortex footer open per range. The main scan now opts small scans out below 10 MiB, as the internal and protected-snapshot scans already do below 256 MiB; Parquet listing scans keep the new default. `small_snapshot_groups_opt_out_of_repartitioning` now asserts the main scan's opt-out and fails with the threshold back at 0. * docs(fork_patches): record duckdb-rs on the protected spiceai-1.4.4-patches-2 head, which table-providers pins * test(duckdb): build CreateExternalTable with DataFusion 55's locations field * ci: validate the datafusion-table-providers pin against spiceai-55, where spiceai#80 landed * test(cayenne): size the point-lookup fan-out control above the main scan's 10 MiB opt-out * fix(data-accelerator-api): implement DataFusion 55's apply_expressions for KeepFirstExec * test: port trunk's new tests to DataFusion 55 (CreateExternalTable locations, common::TableReference) * fix(cayenne): keep -0.0 and 0.0 equal in pushed-down float comparisons and memory-mode indexes DataFusion 55 compares floats with the two zeros equal (apache/datafusion#22835), while Vortex and the Arrow row encoding order them apart. Spell out both zeros when a float comparison or IN list with a zero literal is pushed into a Vortex scan, keep float column-vs-column comparisons above the scan, and hash memory-mode index keys with the file-mode KeyEncoder. * build: move to iceberg-rust 0.11.0 (RC) and opendal 0.58 Points the iceberg crates at spiceai/iceberg-rust#55, which merges the upstream 0.11.x release branch onto the DataFusion 55 fork line. opendal moves to 0.58 to match iceberg-storage-opendal, so one opendal stack is built instead of two. 0.58 removes the OperatorBuilder step: Operator::new returns the operator with the same default layers that finish() applied. * Bump datafusion to 55.2-rc1 * build: pin iceberg-rust at spiceai-0.11.0-df-55 (6317d03e) spiceai/iceberg-rust#55 merged into spiceai-0.11.0-df-55. Same Rust code as the previous pin (e02547cb); the branch adds only a Python CI pin and a doc-comment fix. fork_patches.md records the new branch and that SigV4 is now an AuthManager in the fork's REST catalog. * build: keep the lockfile's unrelated resolutions from the DataFusion bump The previous commit's lockfile update also re-resolved five crates with range requirements (hashbrown, heck, base64). Restore them so the iceberg repin changes only the iceberg-rust source lines. * build: pin iceberg-rust at spiceai-0.11.0-df-55 (7735caa2) Picks up spiceai/iceberg-rust#49 (IcebergTableProvider::catalog and ::table_ident) and spiceai#56 (the fork's own DataFusion pin, which does not affect this build). spiceai#49's fork_patches row and its compile guard land with spiceai#14588, which first calls the getters. * build: pin datafusion at spiceai/datafusion#250 (55.2.0-rc1 and the 54 backports) Moves the DataFusion fork from 02550cf9 (55.1.0) to 1749ce4e, the head of spiceai/datafusion#250, which merges into spiceai#249: `spiceai-55` at 55.2.0-rc1 (spiceai/datafusion#248) plus the `spiceai-54` patches spiceai#233, spiceai#234 and spiceai#245 carried onto 55, and spiceai#250's review fix. The guards for those follow in the next commit. The pin names a pull request branch, so the ledger row is marked TEMPORARY and `scripts/check_fork_patches.py` refuses to let it land until spiceai#250 and spiceai#249 merge and the pin moves to the `spiceai-55` merge commit. The 55.2.0-rc1 merge changes no Spice patch. Outside `Cargo.lock`, the fork's 02550cf9..27351ab0 diff has the same patch-id as upstream's 55.1.0..55.2.0-rc1 diff, except `datafusion-spark`'s `quote.rs`, which the fork's backport had already made identical to 55.2.0-rc1's. That backport's ledger row is dropped; upstream carries the fix as apache/datafusion#25277. `Cargo.lock` changes only the 36 DataFusion entries. * test: guard the DataFusion unparser and footer-cache fixes backported from 54 Each backported fix gets a guard here, so the next re-cut of the fork cannot drop it silently, plus a ledger row naming the guard. - spiceai/datafusion#233 (spiceai#14373): a join that is another join's right input stays a parenthesised joined table on that join's right. `a_join_that_is_another_joins_right_input_stays_on_its_right` checks that `b` is introduced before the outer `ON` names it and that a LEFT JOIN keeps a filter from inside it out of `WHERE`. With the `sqlite` feature it runs the SQL on SQLite and compares the rows with DataFusion's for the same plan. - spiceai/datafusion#234 (spiceai#14375): a `Limit` that is a join input bounds that input, not the join. `a_limit_on_a_join_input_bounds_that_input_rather_than_the_join`, with the same row comparison. - spiceai/datafusion#250: a projection over a join that passes up a mark join's mark is refused rather than unparsed with the mark unbound. `a_mark_a_projection_over_a_join_passes_up_is_refused`. - spiceai/datafusion#245 (spiceai#12952): the file metadata cache frees an evicted footer's hit counter. 55's `DefaultCache` already does, so 55 carries no patch; `crates/cayenne/tests/footer_cache_hit_counter_test.rs` pins the behaviour by counting the live heap across 20,000 cold footer reads into a full cache. At 02550cf9 the three unparser guards fail. SQLite rejects the nested join's SQL ("ON clause references tables to its right"), returns 1 row where the plan returns 2 for `b FULL JOIN (c LIMIT 1)`, and the mark join renders as `… LEFT OUTER JOIN (b) ON a.id = b.id WHERE NOT c.mark`. Against a local DataFusion build with the hit-counter prune removed, the footer-cache guard measures 2,646,176 bytes of heap growth. At 1749ce4e all four pass, and the footer-cache growth is 0 bytes. * build: pin datafusion at spiceai/datafusion#249's merge commit Moves the fork pin from spiceai#250's branch head to bebc4d58, the merge of spiceai#249 into `spiceai-55`. Its tree is spiceai#249's head, cbda233a: 55.2.0-rc1 plus the backported spiceai#233, spiceai#234 and spiceai#245. The pin names a landed revision, so the TEMPORARY marker comes off and `scripts/check_fork_patches.py` passes. spiceai#250 is not part of this revision, so its guard and ledger row are removed. * Update datafusion version to 55.2 in Cargo.toml * docs(fork_patches): record the unparser frame refactor spiceai#249 carries `cbda233a6` is the fourth Spice commit at the pinned revision. It moves the join-input bookkeeping that spiceai#233 and spiceai#234 add out of `select_to_sql_recursively_inner`, so an unoptimised build's frame stays where it was. It had no ledger row, so a re-cut could drop it without the audit noticing. Its row is a GAP, with an Open gaps entry. What it buys is stack headroom in a debug build: without it, the fork's `roundtrip_statement` needs 2,112 KiB and overflows a 2 MiB test thread. This repo runs tests on 8 MiB threads, so no test here sees the difference deterministically. The entry names the fork-side check to run at a re-cut instead. * Fix tests * build: pin DataFusion at spiceai/datafusion#251, so MAX of a partition column skips empty partitions (spiceai#14801) * build: pin datafusion at spiceai/datafusion#251's merge commit Moves the DataFusion fork pin from spiceai#249's merge commit to the head of `spiceai-55`, which adds two patches on top of it: - spiceai#251: a file that holds no rows gets no partition-column min/max. Without it, `MIN`/`MAX` of a partition column is answered from the listing's statistics with the value of a partition whose files are all empty: `max(p)` answers '99' for a `p=99` directory holding one empty file, where the rows give '3'. - spiceai#240: the listing prunes files by metadata-column predicates before opening them and reports those filters `Exact` (spiceai#229 on `spiceai-54`). Neither changes a manifest, so `Cargo.lock` only moves its source revision. Each patch gets a ledger row and a guard in the listing connector's tests, which the gate runs as library tests. * fix(listing): keep every predicate residual on the `_location` fast path The pin now carries spiceai/datafusion#240, so the inner listing reports a metadata-column predicate such as `_size < 50` `Exact`. `MetadataPruningListingTable` handed pushdown classification to the inner listing whenever a `_location` predicate was present, but its `_location` fast path opens the named objects and applies no other predicate, so `_location = x AND _size < 50` returned the rows of a file the `_size` predicate excludes. Report every predicate `Inexact` on that path as well, so a residual `FilterExec` re-applies it. This is spiceai#14790's change, which makes the same fix for partition predicates, taken byte for byte so that whichever of the two lands second merges cleanly. The new test goes through `create_listing_table` with `_location` and `_size` enabled. * test(metadata): expect the listing to answer `_size = 2319` exactly With spiceai/datafusion#240 the listing prunes files by a metadata-column predicate and reports it `Exact`, so the plan for `met_size`, which enables only `_size`, has no residual `Filter`. The scan lists only the four 2319-byte files; `data_0.parquet` is 2317 bytes, as its S3 `Content-Length` shows. Regenerated by running `metadata::s3_metadata_columns` and accepted with `cargo insta accept`. The rest of the test passes unchanged at this pin, including the `_location` plans and the returned rows. * fix(data-connector-api): satisfy pedantic clippy in the metadata-pruning listing test `make lint-rust`'s `--tests` pass rejected two lints in `listing_table_prunes_files_by_a_metadata_column_predicate`: `redundant_closure_for_method_calls` (`|group| group.len()`) and `format_collect` (`map(format!).collect::<String>()`). Use `FileGroup::len` and build the file contents with `writeln!` into one `String`. The test is unchanged in what it checks. --------- Co-authored-by: Luke Kim <[email protected]> Co-authored-by: Sergei Grebnov <[email protected]>
WhaleChao
pushed a commit
to WhaleChao/spiceai
that referenced
this pull request
Oct 12, 2026
…iceai#14844) * feat(cluster): add runtime.executor.task_slots to override executor task slots * feat(cluster): honor runtime.executor on the executor, decouple task-runner threads from task slots * test(cluster): cover runtime.executor.task_slots end to end Start a real scheduler and executor and read the slot count the executor registered with the scheduler, for the scheduler-supplied value, an executor-own override, and the CPU budget default. Link the task_slots validation errors to the runtime reference page and record the datafusion-ballista pin's branch accurately. * test(cluster): guard task-runner pool size against the slot count * ci: run the task-runner thread guard in the integration workflow * revert: drop runtime.executor.task_slots and the executor changes (out of scope for #14845 milestone 1) * feat(cayenne): keep one row per primary key automatically and deprecate on_conflict (#14726) * fix(cayenne): reject duplicate primary keys in full refresh Fixes #14578 * fix(cayenne): honor overwrite dedup modes across batches * fix(cayenne): honor drop conflicts during full refresh * refactor(cayenne): resolve repeated keys with a per-table conflict policy Replace the whole-stream overwrite validation and the shared dedup crate's buffered overwrite mode with a Cayenne-owned policy that follows the documented on_conflict semantics: each incoming record batch is one upsert statement, so a key repeated within a batch follows the dataset's upsert refinement and a key repeated across batches is a later upsert. This commit adds the policy (key_conflicts) and threads the upsert refinement (upsert_dedup / upsert_dedup_by_row_id) from the accelerator into the table and its partitions. The write paths adopt it in the following commits. * fix(cayenne): keep the last copy of a key repeated across a full refresh's batches A file-backed full refresh stored every copy of a primary key its source repeated across record batches (#14578). Under the upsert policies a later batch's copy is a later upsert of that key, so the refresh now resolves it the way separate upserts would: a layer splitter routes each batch into the main snapshot or into protected layers above it, starting a new layer when a batch repeats a key the current layer may hold (or the layer reaches 8M rows), and tombstoning, in each layer above the first, the keys an earlier layer may hold. Both checks use bloom filters charged to the query memory pool; a false positive costs a layer or a tombstone, never a row. The layers, their key tombstones and their thresholds commit in the overwrite's own transaction and publish in its visibility flip, leaving exactly the state an overwrite followed by one upsert per layer leaves. Under drop the first copy is kept with an exact key set. * Resolve repeated keys before buffered Cayenne writes * test(runtime): cover Cayenne on_conflict for keys repeated within and across batches Runs every on_conflict policy through the runtime in mode: memory and mode: file, for full and append refreshes and for a partitioned table, with a key repeated past the first record batch and within one batch. Replaces the narrower Cayenne cases added to the DuckDB-gated on_conflict_options tests. * docs(cayenne): describe how a write resolves keys it repeats * Resolve repeated keys in streaming Cayenne appends atomically * fix(cayenne): size layer key filters to the write and charge appends to their query The layer splitter's first bloom filter held 64K keys (384 KiB at 48 bits) however small the write, and a streaming append charged it to the table's runtime pool rather than the pool of the query running the write. Start at 1K keys and grow each next filter fourfold (eight filters cover a full layer), and register an append's reservation on its task context. * perf(cayenne): keep one tombstone per key for a layered refresh and build its index once A refresh that repeats every key across N layers recorded N-1 tombstones per key and folded each layer into the live deletion index under the listing fence. Tombstones merge by maximum and a layer tombstones only keys it holds, so only a key's highest tombstone matters: keep that one, and build the new snapshot's index outright from the layers before the fence. The per-layer key check uses an exact set of 64-bit key hashes; a chain of bloom filters summed their false-positive rates and split refreshes that repeat no key. Adds a release-mode benchmark (examples/layered_refresh_bench.rs). * perf(cayenne): collapse a bounded window of a streaming upsert write before layering it A streaming full refresh or append now resolves the keys a 128 MiB window of its input repeats in memory (last copy wins) before the layer splitter sees it, so a repeat inside the window opens no layer and leaves no tombstone. Batches are resolved as they arrive, keeping the work per poll to one batch, and the window is charged to the query memory pool. * perf(cayenne): hide a position-deletion table's superseded refresh copies by position On a table that deletes by position, a full refresh whose input repeats keys across its collapse windows now writes every layer into the main snapshot and hides the copies a later layer supersedes with position deletes, found by a read-back of the lower layers' key columns. The refresh publishes one snapshot with no protected layer and no key tombstone, reads skip the deleted pages, and the live row count stays exact. Key-deletion tables keep the protected layers and key tombstones. The read-back lists files without statistics: read before the deletes are installed, they would be cached counting the hidden rows. Position capture casts the key columns Vortex returns in view form to the table's key type, and skips rows a position delete already hides. * docs(cayenne): note that repeated keys are resolved within each partition * perf(cayenne): fold a key-deletion refresh's superseded copies out of its files before publishing A refresh that repeats a key across collapse windows wrote the later copies as protected layers with key tombstones, which made scans and point lookups read every layer until compaction (p99 full scan 51.9 ms, lookup 33.9 ms at 1M keys x 8 copies). Write every layer into the one snapshot instead, locate the superseded copies by a read-back of their keys, and on a table that deletes by key rewrite the files holding them without them. The refresh then publishes one snapshot with no deletes and exact statistics, rebuilt from per-file statistics recorded during the write. A table that deletes by position keeps hiding the copies with position deletes, but its statistics are now served inexact while those deletes are pending: they still describe the hidden rows, and after a reopen MAX() answered from them returned a superseded copy's value. * fix(cayenne): keep the last copy of a key a transaction's write repeats across batches A write inside BEGIN ... COMMIT failed with 'Incoming data contains duplicate primary key across batches' whenever its input repeated a key across record batches, although under the documented on_conflict semantics each batch is one upsert and the later copy wins. Resolve repeats within a batch before validation, validate a repeat across batches as a later copy that supersedes the stored row once, write the layers in order into the staged snapshot, and fold the superseded copies out before the commit. * style(cayenne): format the staged-upsert repeat test * refactor(cayenne): share the layered fold between refreshes and transactions Write the later layers and fold them through one step used by both a full refresh and a transaction's staged snapshot, and derive the live row counts from the statistics rather than threading them through. Pass the cross-batch repeat handling explicitly from the writer that folds, delete superseded files concurrently while releasing their cached footers, log a failed cleanup of an unpublished snapshot directory, and share the NDV sketch merge. * perf(cayenne): locate superseded refresh copies as they are written Route every layered write through one exact map from each admitted key's 128-bit digest to its layer and, once the write observer sees it land, its file and position. A later layer admitting the key again records the earlier copy's position directly, so a refresh no longer reads its lower layers' keys back to find the copies they supersede, needs no bloom filter or per-layer key set, and no longer cuts layers at a row cap. The map is pre-sized from the replaced table's row count and charged to the query memory pool; a row the observer cannot locate fails the refresh. * perf(cayenne): skip the collapse window's filter when it holds no repeated key A window over distinct keys looked every row up again on drain only to keep all of them; it now passes its batches through when no key repeated. At 40M distinct keys this halves the window's cost (0.78 s to 0.41 s). * fix(cayenne): resolve repeated keys only for refresh writes A dataset's on_conflict now resolves the keys a refresh's incoming data repeats; a user's statement keeps its own semantics. The refresh task marks the session it writes through (util::session_state::AcceleratorRefreshWrite), and Cayenne's insert_into resolves repeats only for a write that carries it, so a user INSERT or a write inside BEGIN ... COMMIT behaves as before: a key it repeats across record batches fails it. Change streams, which do not go through insert_into, keep change-stream semantics. * fix(cayenne): mark user statements instead of refresh writes A refresh path that did not mark its session (a localpod parent's MultiSink, say) fell back to storing every copy of a repeated key. Mark the user's statements instead — AcceleratedTable::insert_into and the write-back sink, via util::session_state::UserStatementWrite — and resolve repeats for every other write, so a missed path deduplicates rather than duplicates. Partitioned inserts carry the marker to each partition's table through PartitionContext. * fix(cayenne): bound the collapse window by everything it holds The window filled at COLLAPSE_WINDOW_BYTES of rows but also charged the map of their keys, uncounted, and kept that map's capacity across drains: an 8 MiB window peaked at 16.2 MB and held its map between windows. Count the map toward the bound and release it on every drain. * fix(cayenne): spill a refresh's key map instead of failing when the pool refuses it A refresh with on_conflict reserved query-pool memory for every key it admitted and failed the whole refresh when the pool refused, even with no key repeated (1M keys in a 64 MiB pool, 16M in 1 GiB). Now a refused reservation cuts the layer at the next batch; between layers, with every admitted key located, the map spills to the runtime's spill directory as a run sorted by digest, and after the last layer the runs merge with the map to find the superseded copies (highest layer wins; lowest under drop). drop takes the same path in place of its unbounded exact set, and the collapse window drains early rather than fail when the pool refuses it. * revert(accelerator-cayenne): keep the auto-tuned config log as on trunk An earlier commit on this branch added metastore_dir to the auto-tuned config log line and its re-emit fingerprint, which is unrelated to resolving repeated keys; drop it. Also move upsert_dedup_for out from under scan_view_reuse_for's doc comment. * fix(cayenne): resolve the repeats a partitioned append refresh holds within a window A partition's append is one staged snapshot of the cross-partition append, so it cannot take the layers a streaming append splits a repeated key into, and a key its data repeated across record batches failed the refresh. Put the bounded collapse window in front of it: repeats within a window resolve per on_conflict, and a key repeated further apart is still rejected by validation, as before. Document the remaining limitation and cover the partitioned append refresh in the runtime matrix. * docs(cayenne): describe the window a partitioned append refresh resolves repeats in * fix(cayenne): keep the last copy of a key a refresh repeats within one batch under upsert Conflict validation already keeps the last copy of a key repeated within a batch under a plain `upsert` (`UpsertOptions::last_write_wins`), so an append refresh of such data loaded on trunk. Resolving the refresh's repeats ahead of validation rejected them instead, failing loads that used to succeed. A plain `upsert` now resolves as `upsert_dedup_by_row_id` does: the last copy wins within and across batches, so a refresh's outcome no longer depends on where record-batch boundaries fall. `upsert_dedup` still fails on differing copies within a batch. * chore(cayenne): drop stale fingerprint tests and fix lint on the layered refresh path Remove the auto-tuned config fingerprint tests that still passed a metastore path after that change was reverted, take the map entry once when keeping a key's first copy, make CollapseWindow::push infallible, and pass the partitioned append's user-statement marker through a builder method. * fix(runtime-table): mark a user UPDATE as a user statement UPDATE writes its new rows through the accelerator's own insert, which never passed the marked AcceleratedTable::insert_into, so Cayenne resolved the keys it repeated as it does for a refresh: an UPDATE moving rows from several batches onto one key succeeded and kept one row instead of failing as on trunk. Mark the session once, before choosing a write path, in both insert_into and update. * feat(acceleration): add on_conflict upsert_dedup_by_time_column Keep the row with the greatest time_column per primary key across an acceleration refresh, so out-of-order source data (late files, Iceberg snapshots with older rows) no longer overwrites a newer version. A streaming step in the refresh keeps key -> greatest time and writes a row only when it is strictly newer than every version already kept, collapsing a key's versions within a batch. On append, it is seeded from the acceleration's keys and times inside the refresh_append_overlap window, replacing the unchanged-row check for this mode. Superseded rows are counted in dataset_acceleration_refresh_rows_superseded{reason=older|equal_time}; on append the count includes rows re-read from the overlap window. A NULL time_column fails the refresh with a count, never row values. Available on Cayenne and SQLite, which apply a key repeated across one write's batches in order. DuckDB applies a write as one INSERT ... ON CONFLICT and keeps an unspecified copy of a repeated key, so it is refused at load, along with a missing time_column or declared primary key, a time_column inside the key, and refresh modes other than full and append. Warn when an append has no refresh_append_overlap. * test(runtime): satisfy pedantic lints in the on_conflict batch tests * test(acceleration): integration tests for upsert_dedup_by_time_column Cayenne and SQLite, in memory and file mode: a full refresh over two files whose newest versions of two keys arrive in opposite orders, a key repeated within a batch, and a filler spanning several batches; an append where a late, older row inside the overlap must not replace the stored version and a newer row must; and a NULL time_column failing the refresh. With plain on_conflict: upsert all three fail. * refactor(cayenne): share the collapse pump and move spill work off the async thread - One Collapser resolves input batches and a window's repeats for both the layer source and the partitioned append's CollapseStream. - The survivor comes from the resolver's policy (Survivor::for_policy); the window takes it instead of a separate keep-first flag. - A drop overwrite takes no window: the splitter already drops a later copy as it admits it. - An overwrite's spill directory is part of its split mode, so there is no unspillable overwrite to fail. - An append clears its map when a routed batch starts a layer, so the admitted-layer field goes. - Spilling and taking the superseded copies swap the map under the lock and sort, write, merge and bucket on the blocking pool; merge sources are iterators, and spill I/O uses 1 MiB buffers. - for_user_statement takes the marker, so each partition sink is one call. - Docs: drop's path, the spill, and the window's users. * proto(cayenne): resolve a full refresh's repeated keys after the write Prototype, behind SPICE_CAYENNE_REFRESH_DEDUP=postpass: a full refresh writes once into its new snapshot, each batch resolving its own repeats (a vectorized hash check first, the exact resolve only for a batch that may repeat a key) and stamping its rows with the batch's arrival sequence in a trailing u32 column the table schema does not name. After the write, a query over the files' key columns, ordinals and row positions, run in hash chunks of about 1 GiB of written data, finds every copy other than the one the policy keeps; position mode hides them with position deletes, key mode folds them out of the files that hold them. The layered refresh bench gains --key-kind (int, string, composite), --dup-fraction, --progress-rows, --memory-limit-mb, --skip-queries and --sql. * proto(cayenne): resolve a streaming append's repeated keys after the write Prototype, behind SPICE_CAYENNE_REFRESH_DEDUP=postpass: an unpartitioned streaming append writes once into a new protected snapshot, each batch resolving its own repeats and stamping its rows with its arrival sequence. Conflict validation still supersedes (upsert) or keeps (drop) the stored copy an incoming key meets, but keeps a key the append repeats across batches; only the key's first copy records the stored copy, so the live row count counts it once. After the write, the duplicate query over the new snapshot's files alone finds the incoming copies the policy does not keep: position-deletion tables hide them with position deletes published with the snapshot, key-deletion tables fold them out of the files. A write of one batch skips the query. The layered refresh bench gains --append. * proto(cayenne): size the post-pass duplicate query's chunks by rows written The query's memory follows the rows a chunk covers, not the bytes they were written in. A well-compressed table packs many rows into each GiB written, so chunks sized from bytes alone held tens of millions of rows each. Chunks now cover at most 16 Mi rows (override: SPICE_CAYENNE_REFRESH_DEDUP_CHUNK_ROWS), and still at most one CHUNK_BYTES of written data. * proto(cayenne): chunk a refresh's duplicate query by written key ranges Behind SPICE_CAYENNE_REFRESH_DEDUP_CHUNKING=range, the post-pass reads each written file's footer bounds on the key columns, groups files whose bounds overlap, and gives each step of the duplicate query only its own files (hash-slicing a group larger than a chunk), instead of reading every file for every hash chunk. Overlapping files fall back to the hash chunks. SPICE_CAYENNE_REFRESH_DEDUP_SPLIT=1 splits a step's files into row ranges when it has fewer files than partitions. SPICE_CAYENNE_POSTPASS_DEBUG=1 prints per-file bounds, the plan, per-step timings and the pool peak. The bench gains --key-order sorted. * proto(cayenne): split a large key-range cluster into key sub-ranges pushed into the scan Behind SPICE_CAYENNE_REFRESH_DEDUP_SUBSPLIT=range: a cluster of files larger than a chunk is cut into equal-width sub-ranges of an integer key column, each step reading the cluster's files with a key-range filter the scan prunes zones by, rather than hash slices that each read the whole cluster. Also passes the written row count to the streaming append's duplicate query after the merge. * proto(cayenne): cut key ranges at sampled quantiles for non-integer leading keys Key sub-ranges become half-open [lo, hi) bounds of any key type that tile the whole domain. A cluster on an integer column is cut equal-width from its bounds; one on another column (strings) at quantiles of about 1,024 evenly spaced rows sampled per file. Each split cluster checks that its ranges read every one of its footer-counted rows, and fails the refresh otherwise. * refactor(cayenne): resolve a refresh's repeated keys after the write, by default Make the post-write resolution the only path a refresh that resolves repeated keys takes: a full refresh, and a streaming append on a table without a partition column. The duplicate query always chunks by the written files' key ranges, cutting a large cluster into key ranges pushed into the scan, sized from the rows and bytes written. Remove what it replaces: the overwrite's key map, its write observer, the spill and merge, the layered overwrite and append, and FirstCopyFilter. The collapse window, still used by a partition's staged append, moves to collapse_window.rs, and Survivor to key_conflicts.rs. Remove the prototype switches and diagnostics. The repeated-key tests move to repeated_keys.rs and gain a refresh cut into many key ranges for integer and string keys. * wip(acceleration): upsert_dedup_by_time_column content-hash ties, query-pool spill, Arrow support Settle equal times by a 64-bit content hash, charge the key map to the query memory pool and spill when it is refused, show the mode's refresh errors unwrapped, keep the last copy of a repeated key on Arrow for this mode, and refuse an on_conflict that is not the primary key or a partitioned Arrow acceleration. Includes threading the query RuntimeEnv into refreshes and a warning for upsert with refresh_append_overlap. * test(cayenne): a staged append refuses a keyed table Dual-write reaches Cayenne only through begin_staged_append, which refuses a table with a primary key in either deletion mode, so a user statement through dual-write never reaches the repeated-key resolution a refresh runs. * fix(cayenne): store a refresh's arrival column under a name the table does not use A table with its own column named __cayenne_arrival failed every refresh that resolves repeated keys ("StructLayout must have unique field names"). The arrival column is now stored under the first of __cayenne_arrival, __cayenne_arrival_1, ... the table does not already use, and the read-back reads it by that name. * wip(acceleration): resolve upsert_dedup_by_time_column full refreshes in Cayenne's post-pass A full refresh of an unpartitioned file-mode Cayenne table hands each row's version (time, content hash) to the write instead of resolving repeated keys in a map: Cayenne stamps them in hidden columns as it writes, keeps each key's greatest version within a batch, and its post-pass keeps each key's greatest (time, hash, arrival) across batches. Other engines and append keep the selector. Every refresh write reports the copies of repeated keys the accelerator does not keep, so rows_superseded covers every on_conflict mode on Cayenne, and rows_written counts every row a refresh receives on every path. * wip(cayenne): order a buffered overwrite's repeated keys by row version A full refresh small enough for the inline path now keeps each key's greatest (time, hash) rather than skipping that path: the buffered rows are sorted by version before the existing collapse keeps each key's last copy, and the copies not kept are reported. * wip(acceleration): resolve upsert_dedup_by_time_column versions before a synchronized child sees them - A dataset with a synchronized (localpod) child resolves row versions in the runtime before broadcasting, since a child cannot read them; MultiSink refuses row versions if a child attached mid-refresh. - MultiSink propagates a table's write error instead of discarding it. - An append seed with a string time column within 14 hours of 1970 reads every stored row, since the widened window has no earlier start. - Content hashes normalize -0.0/NaN only for SQLite, which stores them that way; elsewhere an equal hash means identical rows, so the kept copy is independent of arrival order. - Group the refresh's dedup settings into UpsertDedupByTimeColumn; simplify the superseded-metric reporting and the post-pass handoff. * test(cayenne): pass the overwrite's delete files in the turso statement-conflict test Trunk's statement-conflict test calls commit_overwrite with the three arguments it took before this branch added the delete files a refresh commits with its snapshot. * perf(cayenne): load a refresh's first append into an empty table as one unchecked write A refresh appending into a keyed table that holds no rows has no stored key to check its own against. Under the table's write lock, the sink observes every place a row can sit (data files, cold tier, inline rows, the in-memory tier, staged appends) and, when all are empty, writes the load as one append: no segments, no conflict check, and therefore no in-memory set of every incoming key. The keys the load repeats are still resolved after the write, and it publishes as any append does (one protected snapshot), so the table's shape is unchanged. The load records no keys, so it drops any cached primary-key index it did not fill and warms a new one in the background for the next refresh. A user's statement never takes this path. Cayenne benchmark, 100M distinct keys into an empty table: 65.3 s at 6.8 GiB peak RSS (segmented append) becomes 11.2 s at 3.8 GiB. Tests that build their state with plain inserts now mark them as user statements, which they are, so they keep the append path they assert on. * fix(cayenne): a refresh keeps the last copy of a key under every upsert policy Under `upsert_dedup`, a refresh failed when one record batch held two different copies of a key, while the same copies in two batches loaded: whether a refresh succeeded depended on batch boundaries the user cannot see. A refresh now keeps the last copy under `upsert`, `upsert_dedup` and `upsert_dedup_by_row_id` alike, within and across batches, on every path (full, append, partitioned, memory mode, change streams). The accelerator's `UpsertDedupTableProvider` checked each batch before Cayenne saw it, so Cayenne's resolution never ran for those copies. Cayenne now wraps with `wrap_with_statement_upsert_dedup_if_needed`, which deduplicates only a user's statement and passes the accelerator's own writes through. A user's statement keeps statement semantics. Rollback tests that used the old failure as their trigger now use a null primary key, which fails every policy. * perf(cayenne): share the duplicate query's memory budget between concurrent refreshes Each write that resolves its repeated keys after the write sized its duplicate query's chunks for a full budget of its own (16M rows, about 1 GiB). The partitions of a partitioned table resolve their repeats at the same time, so a partitioned full refresh held one full chunk per partition at once. The chunk budget is now shared: a write claims a share when it starts and frees it once its repeats are resolved, and each sizes its chunks for its share. A single refresh keeps the whole budget; eight partitions take an eighth each. Measured with the chunk size swept by hand on an 8-partition full refresh of 100M keys plus 20% repeats: 16M-row chunks peaked at 9.9 and 11.0 GiB, 1M-row chunks at 4.1 and 4.3 GiB (trunk: 3.8-4.4 GiB), with no slower load; an unpartitioned refresh keeps its full-size chunks, which smaller ones would slow by about 12%. * wip(cayenne): order a refresh's first load into an empty table by row version A refresh that supplies row versions (`upsert_dedup_by_time_column`) and appends into an empty table now takes Cayenne's one-write first load with the version columns stamped, and resolves the keys it repeats by version instead of arrival. Any other append that carries row versions is refused: it would supersede a stored copy whatever its version. The runtime hands versions to an append only into an acceleration it observed empty with no retention filter; otherwise its selector resolves them before the write. The first-load integration test now also runs `refresh_mode: append` on every engine, with enough rows that the load streams in many batches. * Integrate snapshot metadata listing with the incremental secondary index * Account for retained key digests and queued collapse batches * Add Cayenne per-statement conflict-policy and failed-append regressions * Apply strict Cayenne upsert semantics across incoming batches * Guard failed Cayenne snapshot publication against durable partial updates * Commit Cayenne append deletions and snapshot visibility together * Align Cayenne lifecycle regressions with explicit conflict policies * Resolve staged Cayenne statements across the whole input * Compare Map and nested dictionary payloads in Cayenne content identity Strict upsert collapses identical copies by a digest of each row's Arrow row encoding. The row format cannot encode Map or dictionaries of nested values, so any table with such a column failed every keyed write. Rewrite those columns as the value IS NOT DISTINCT FROM compares (a map as its entry list, a dictionary as its values) before encoding. A seeded differential test checks the digest against DataFusion's IS NOT DISTINCT FROM on every pair of rows, per column and per row, including signed zero, NaN payloads, NULLs over masked children and repeated dictionary values. The helper-column layout test and the name-collision test now cover both content columns. * Resolve a Cayenne append's failed COMMIT by reading back its outcome A COMMIT that reports an error may still have committed. The resolved append then deleted its snapshot directory and deletion vectors while the catalog referenced them, and the table failed to reopen with a missing deletion vector. Read back the snapshot's sequence row, which commits with the rest of the publication: present, publish; absent, discard; unreadable, keep the files and refuse further writes until the table is reloaded. Tests inject each outcome into the commit and pause a commit to cancel its caller, which must not release the write lock before publication. * Count every Cayenne key with different versions in the whole statement Strict upsert stopped at the first record batch that held different versions of a key, so the count in its error depended on batch boundaries (2, 3 or 4 for the same rows). A buffered write now checks all of its rows before failing, and a streamed write splits a batch's versions of a key into separate batches so the post-write resolution counts each key once across the statement. * Let Cayenne resolve every write's repeated keys itself Cayenne applies on_conflict to refreshes, change streams and user statements alike, so the factory no longer wraps it in the generic statement deduplication provider, and the session marker that told the two kinds of write apart is gone. * Resolve a Cayenne partition's append over its whole input A partition's staged append resolved repeated keys only inside a bounded in-memory window and rejected a key repeated further apart, so whether a refresh succeeded depended on how far apart its batches repeated a key. It now stamps arrival and folds the superseded copies out of its staged files with the same post-write resolution a streaming append uses, and the window is removed. * Resolve Cayenne transaction statements whole and read back a failed commit A multi-table transaction now publishes each table's statement resolved over its own input, skips a table it wrote no rows to instead of publishing an empty snapshot, and reads back each staged snapshot's sequence after a COMMIT reports failure, keeping the staged files when the outcome cannot be determined. Tests drive the real transaction orchestrator across two tables in both deletion modes and inject each commit outcome. * Add upsert_by_arrival and count the rows Cayenne refreshes supersede upsert_by_arrival keeps the version of a key that arrived last; it loads only on Cayenne and other engines refuse it by name. On Cayenne the deprecated aliases upsert_dedup and upsert_dedup_by_row_id warn once at load and name the option they behave as. The strict upsert error no longer mentions an option this release does not have. dataset_acceleration_refresh_rows_superseded{reason} counts the rows a refresh received but did not keep: unchanged for an identical copy, arrival for a version settled by arrival order, including a row drop discards for a stored key. Cayenne counts them into a SupersededRows the refresh attaches to its session; the reasons an option can produce are published at zero at load. A localpod parent's refresh now reports its write's failure instead of succeeding with no change. * Align Cayenne integration tests with the conflict policies Plain upsert now fails on different versions of a key in one statement, so the keep-last tests use upsert_by_arrival and a new test covers the strict failure. A first insert into an empty table builds its key index in the background, which the accounting tests now wait for. * Satisfy rustfmt and the pedantic test lints * Refuse deletes and table maintenance once a write's commit outcome is unknown A table that could not learn whether a write's catalog commit happened refused further appends and staged writes, but statement and retention deletes, MERGE key-probe deletes, inline and mem-tier checkpoints, compaction and cold-tier promotion still ran from its in-memory view, which may miss rows the catalog holds. Each now checks the same guard after taking the write lock, so only reloading the table from its catalog lets it change again. * refactor(cayenne): align conflict policy names with on_conflict semantics * test(cayenne): scope CDC replacement assertions to upserts * Retry the duplicate query with fewer partitions once it runs out of memory The post-write duplicate query retried a ResourcesExhausted failure only by cutting the key space finer. Each partition's hash join and aggregate still reserve a fixed minimum (about 320 KB per join input), so with many partitions and a small pool no chunk size fits, and a retry that started while the failed attempt's tasks still held their memory found the pool full. On a 24-core host the 8 MB-pool test failed 3 of 4 tries in signoff. Each retry now also halves the query's partitions, down to one, and first waits (up to 10 s) until the failed attempt has released the memory it held, counted by a pool wrapper scoped to the query. Under SPICE_CPU_CORES=64 with every core loaded, the bounded-pool test failed 1 of 5 runs before and 0 of 10 after; waiting alone left 5 of 10 failing. * Adapt time-ordered refresh spill files to DataFusion 55.1 DataFusion 55.1 returns spill files as Arc<dyn SpillFile>: write through open_writer(), which accounts disk usage as it writes, and read back by the file's local path. The Arrow IPC writer over a SpillWriter is not Sync, so the deferred-rows writer sits behind a mutex reached only via get_mut(). TableReference moved from datafusion::sql to datafusion::common. * Keep the last arrival for every Cayenne table with a primary key #14576: a primary key alone decides what Cayenne does with a key it sees again, so on_conflict no longer chooses between versions. - A refresh or statement keeps the copy of a repeated key that arrived last, within and across batches. The failing `upsert`, `drop`'s first-copy rule, and the row-content digest that told identical copies from versions are gone, with the UpsertPolicy plumbing that selected between them. - A table opened with any on_conflict upserts on its primary key, so a table an earlier release created with `drop` (DoNothing) keeps the last version too. pk_conflict_detection: none still leaves a table without conflict resolution. - Cayenne's table creation ignores the on_conflict values and their validation. The setting stays on the acceleration because it still keeps a read-write dataset's writes in the acceleration. - Every copy a write supersedes counts as `arrival`, seeded at zero for a dataset with a primary key. * Order a key's versions by time_column whenever one is set #14576: a Cayenne acceleration with a primary key and a time_column keeps the newest version of each key by that column, without an on_conflict value asking for it. - A full or append refresh orders versions by time_column whenever the dataset sets one. It keeps the last arrival instead when the table has no primary key, its key holds the time column, or the rows the refresh reads lack it, so no configuration that loads today is refused or fails. - Equal times go to the later arrival, so a correction that keeps its time replaces the row it corrects. A row identical to the kept one is a re-read: it is not written again, and counts as superseded by arrival. - A NULL time is older than any time rather than failing the refresh. - Versions carry only their time: the content hash Cayenne ordered ties by, and its trailing column, are gone. Superseded rows count as `older` or `arrival`. - The load-time refusals of `upsert_by_time` are gone with the field that enabled it; the refresh's internal names follow the rule it implements. * Remove the unreleased upsert_by_arrival and upsert_by_time values #14576 replaces explicit on_conflict values with a rule inferred from primary_key and time_column, so the two values this branch added never ship: on_conflict is back to drop, upsert, upsert_dedup and upsert_dedup_by_row_id, with the error that refused the new values off Cayenne. The warnings that named them go too: the one calling upsert_dedup and upsert_dedup_by_row_id aliases of the new values, and the one recommending upsert_by_time for an append that re-reads an overlap. The no-overlap warning now describes the inferred rule instead of the value. * State each Cayenne dataset's key rule at load and deprecate on_conflict #14576's load lines: - A Cayenne dataset with a primary key logs the rule it keeps one row per key by: the newest by time_column (the later arrival on a tie), the last arrival (noting that it can differ between refreshes, and that a time_column makes it reproducible), or each change in order for refresh_mode: changes. - A dataset that sets on_conflict is warned that it is deprecated. On Cayenne the warning says it is no longer used and names any change in which version a key keeps; on another accelerator it says on_conflict is removed in 3.0 and that Cayenne keeps one row per key without it. A read-write dataset is not warned: on_conflict still keeps its writes in the acceleration. - The Spicepod schema marks on_conflict deprecated. * Report rows a refresh superseded as dataset_acceleration_rows_superseded A Cayenne table keeps one row per primary key, so the rows a refresh receives but does not keep are counted under one `dataset_acceleration_rows_superseded` counter with the two reasons #14576 specifies: `older` (a later time_column was kept) and `arrival` (a later arrival was kept). Counting a user's statements follows separately. * Describe Cayenne's primary key rules in the technical reference The repeated-keys section still described the on_conflict policies, the content digest, and the refresh_rows_superseded metric. State the #14576 rules instead: last arrival, newest by time_column, change order; and dataset_acceleration_rows_superseded with its older and arrival reasons. * Let a memory-mode Cayenne table order a full refresh's versions itself A memory-mode full refresh of a dataset with a primary key and time_column ran the refresh's own newest-by-time selection over every row, then Cayenne resolved the same keys again over the buffered write. Cayenne's buffered write already orders a key's copies by the row versions a refresh supplies, so a memory-mode overwrite now gets the versions, as a file-mode one does, and the refresh no longer selects. An append into an empty table still selects in memory mode, which only file mode resolves after writing. Release spiced, 5M rows over 2.5M keys, memory mode, full refresh, 8 refreshes per run, arms interleaved, same rig: P99 681 ms -> 389 ms (P50 592 -> 353 ms); the selection was 35.8% of busy CPU in a profile of the old path. File mode runs the same code in both arms (P99 840 vs 857 ms). Every run kept exactly one row per key, the newest. * Count the rows a statement supersedes, and pin its row count #14576 counts a user's statement under dataset_acceleration_rows_superseded as well as a refresh. The rows a statement's write does not keep are counted into its session and recorded by the accelerator write lock once the write completes; a failed write records nothing. A statement's row count reports rows inserted or replaced, as PostgreSQL's does. Cayenne's write already counts that way: a copy of a key that a later copy in the same INSERT replaced is not counted, and a row replacing a stored key is. Pin it in memory and file mode. * Fail an UPDATE that gives a row a key another row keeps An UPDATE that reassigned a Cayenne table's primary key ran as a delete and an insert, so a row moved onto a stored key silently replaced it, rows moved onto one new key collapsed to one row, and a NULL key deleted the row before the insert rejected it. Per #14576 such an UPDATE now fails and changes nothing, as in PostgreSQL: Failed to update dataset 'events': the new 'id' of 1 row matches a key already stored, so nothing was changed. UpdateExec materializes the updated rows once and checks those exact rows with the table's UpdateValidator before deleting anything; Cayenne's validator refuses a NULL new key, a key two updated rows share, and a key a row the statement leaves in place holds. A partitioned table prepares every partition's update before writing any, so a refusal in one partition changes none. * Describe the UPDATE key check and statement counting in the technical reference * Write an overwrite's position deletes into the snapshot it publishes An overwrite that repeats keys on a position-deletion table hid the superseded copies with deletion vectors written under the table's current snapshot, the one the overwrite replaces. That directory is then retired: its deletion vectors keep it alive, a failed overwrite (which removes only its own directory) strands them, and the wholesale reclaim that runs when a table has no snapshot manifest would remove deletion vectors the published snapshot still references. Write them under the snapshot the overwrite publishes. Through spiced, a file-mode full refresh repeating keys referenced four delete files under the previous snapshot directory; the new test fails before this change with "delete file .../<replaced>/deletions/... is outside the published snapshot". * Say when dataset_acceleration_refresh_rows_written and rows_superseded are counted rows_written counts rows as a refresh reads them, and rows_superseded once the refresh succeeds, so a refresh that fails is counted in the first only. The description no longer implies the two subtract to the rows kept. * Format the overwrite position-delete test * Satisfy pedantic lints in the UPDATE and buffered-version paths * Keep a read-write dataset's writes in the acceleration with write_mode: acceleration Any on_conflict value used to keep a read-write accelerated dataset's writes (and its transactions) in the acceleration, outside a change stream, and let its source be read-only. That routing was never documented and tied write placement to a conflict setting #14576 deprecates. A new `acceleration.write_mode: acceleration` selects it instead, and on_conflict no longer routes writes: - write_through (default) and write_back still need a source that accepts writes; acceleration does not. A sink source is unchanged. - A read-write dataset over a read-only source without it fails at load, naming the fix; write_mode: acceleration with refresh_mode: changes fails too, since the source's changes would overwrite the writes. - The on_conflict deprecation warning now reaches read-write datasets. With on_conflict routing removed, 24 integration tests over read-write file-sourced Cayenne datasets (DML, transactions, the UPDATE key check) failed to load; with write_mode: acceleration set they and the rest of the affected suites pass (54/54). * Build the partitioned UPDATE fixture without format_push_string * Refuse write_mode: acceleration with a connector's default change stream The check read the literal refresh_mode, so a cdc: dataset that omits it, and refreshes by changes by its connector's default, loaded with write_mode: acceleration and let the change stream overwrite its writes. It now reads the connector-resolved mode, and the message says the dataset refreshes by `changes` rather than quoting a setting it may not have. Through spiced, a cdc: dataset with write_mode: acceleration and no refresh_mode loaded before (count 0) and is refused after. write_mode_acceleration_is_refused_with_a_default_change_stream (needs the debezium feature) fails with the literal check ("got Some(Ready)") and passes with this one. * fix(runtime-table): keep the overlap dedupe for append refreshes that keep the last arrival An append refresh on a Cayenne table with a time column but no usable primary key skipped the exact-row dedupe, so every refresh re-appended its whole refresh_append_overlap window. * fix(runtime): seed superseded-row metrics from the connector-resolved refresh mode A connector whose default refresh mode is changes, with refresh_mode omitted, was seeded as full and published an arrival series that its refreshes never report. --------- Co-authored-by: Phillip LeBlanc <[email protected]> * fix(object-store): stop reading a response body at its first error (#14831) object_store's SpawnedReqwestConnector forwards response-body frames from its I/O task until the body ends. Once a request outlives client_timeout, reqwest's body returns a new timeout error on every poll and never ends, so the task pushes errors into an unbounded channel until object_store drops the body after its retry backoff and retry request. On a ~5 MB/s link, loading the 195 MiB TPC-H lineitem file from S3 grew RSS past 25 GB. Add object_store_spawn::SpawnedReqwestConnector, which runs request I/O on the given runtime the same way but stops after forwarding the first body error, use it at every call site, and disallow the upstream constructor. Fixes #14830 * fix(ci): use byte-order collation in the benchmark Postgres container (refs #14815) (#14836) * fix(listing): keep a partition predicate as a residual filter on the `_location` fast path (#14790) `MetadataPruningListingTable::scan` takes a fast path when the query carries a `_location` predicate: it opens exactly the named objects and applies only `_location` and `_last_modified`. It applies no other predicate. `supports_filters_pushdown`, however, only reported `Inexact` for the `_last_modified` prune, and explicitly fell through to the inner listing whenever a `_location` predicate was present. The inner `ListingTable` reports a partition-column predicate `Exact`, so DataFusion dropped it from the plan while the fast path never applied it. A query combining `_location` with a partition predicate then returned rows the predicate excludes. Report every filter `Inexact` whenever either path applies, so the residual `FilterExec` re-checks them while the prune still removes the files that cannot match. The four `_location` EXPLAIN snapshots are unaffected: each filters on `_location` alone, and for a lone `_location` filter the inner listing already answers `Inexact` (their committed plans carry a `Filter` above a `partial_filters` scan), which is what this now returns directly. Co-authored-by: grokspice <[email protected]> Co-authored-by: Emma <[email protected]> * fix(ci): search only the unixodbc keg, not all of Homebrew's lib, in the macOS release builds (#14846) `RUSTFLAGS=-L $brew_prefix/lib` is searched before a build script's own link-search path, so a static library Homebrew has linked into that directory under the same name replaces the vendored copy when rustc bundles it. On a runner holding a foreign libcrypto.a, openssl-sys bundles that instead of the vendored OpenSSL librdkafka was compiled against, and the spiced link fails on undefined _ENGINE_* symbols. The odbc build only needs libodbc, which the unixodbc keg's own lib directory provides; setup-cc already scopes every other macOS job the same way. * fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC (fixes #14753) (#14840) * fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC The generic-driver routes installed no SQLite expression gate, so a timestamp literal federated as CAST('...' AS TIMESTAMP), which SQLite evaluates to the year as an integer (every row matched), and an interval as INTERVAL '1 HOURS', which SQLite cannot parse. The sqlite engine now gets the SQLite connector's gate plus a refusal of every temporal value. Fixes #14753 Fixes #14754 * test: simplify the SQLite temporal-gate regression tests * docs: say why the sqlite engine has a federation gate * test(odbc): the SQLite profile now keeps unfaithful casts and temporal values local * test(odbc): give the SQLite temporal checks their own test, with a date literal * fix(odbc): apply the federation policy to the scan's filter pushdown too ODBCTableFactory built its SqlTable without the function-support policy, so when the federation analyzer refused a plan, the filter fell back to the scan's own pushdown, which consulted no policy. On the ODBC SQLite profile a timestamp filter was pushed as CAST('...' AS TEXT) and returned every row. --------- Co-authored-by: t <[email protected]> * chore: pin the fork at spiceai/datafusion#249 and guard the DataFusion fixes backported from 54 (#14800) * build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a85; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing #237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for #237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (#70, #71, #72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (#82, #53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, #54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, #15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, #16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, #7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, #50), datafusion-table-providers 465926a3 (spiceai-55, #80), snowflake-rs f5557381 (spiceai-59, #13), spark-connect-rs 18ae9bd3 (spiceai-59-2, #15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: #49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. #68 is still open, waiting on license-compliance decisions. Pin 718429f, the #73 merge, until #68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, #73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of #68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 7184…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges upstream Ballista
main(da205206, the DataFusion 55 line; there is no 55 release tag) into the Spice line (spiceai-55, cut fromspiceai-54). The merge commit message records what was taken from upstream (patches upstream adopted) and what Spice keeps.On top of the merge:
vortexshuffle moves to the current Vortex Arrow APIs; vendored DataFusion protos, config docs, rustfmt/taplo and machete are brought in line, so the Ballista Rust CI is green.ballista.planner.adaptive.enabledstaysfalse: the fork cannot persist adaptive graphs.Pinned by spiceai/spiceai#14612 (tracking: spiceai/spiceai#13570). Supersedes #66.