Merge upstream DataFusion 55.1.0 into spiceai-55 by krinart · Pull Request #243 · spiceai/datafusion

krinart · 2026-10-02T00:41:04Z

Merges upstream DataFusion 55.1.0 into the Spice line (spiceai-55, cut from spiceai-54); conflicts are resolved in the merge commit and recorded in its message.

On top of the merge:

Join statistics no longer carry input sum_values through inner and outer joins, so SUM(col) over a join is not answered from statistics with the input table's sum (fix(physical-plan): drop input sums from inner and outer join statistics #239).
A Date32 literal is unparsed with the dialect's date cast type, so a date range pushed to SQLite keeps its rows (fix(unparser): spell a Date32 literal's cast with the dialect's date type (refs spiceai/spiceai#14491) #237, via Merge the Date32 literal fix (#237) into spiceai-55-patches-3, without metadata-column pruning #242).
datafusion-spark quote.rs imports its expression types from datafusion_expr, so the crate builds without its core feature (55.1.0 has the broken import).
The array_any_value empty-element guard is dropped: 55.1.0 carries the same guard.
The ORDER BY unprojection no longer uses an if let match guard, which broke the workspace MSRV (1.94).
CI lint (clippy, rustdoc, rustfmt, taplo) is clean; builds against the arrow-rs fork at db730b42.

The metadata-column listing pruning from spiceai-54 (#229) is not carried on this line yet, because Spice has no test that fails without it.

Please merge with a merge commit, not squash, so each patch stays its own commit.

Pinned by spiceai/spiceai#14612 (tracking: spiceai/spiceai#13570). Supersedes #238.

…che#22980) ## Which issue does this PR close? * Part of apache#20835 ## Rationale for this change `FixedSizeList` containing `Struct` values was not handled by the existing recursive nested adaptation logic used for schema evolution. As a result, planner-time compatibility checks, nested cast detection, and runtime casting did not support additive struct evolution within `FixedSizeList` containers. This change adds `FixedSizeList` support and verifies planner/runtime parity so that planning allows exactly the cases runtime can adapt while continuing to reject incompatible schema changes. ## What changes are included in this PR? * Extend `cast_column` to support recursive casting of `FixedSizeList` values when source and target list sizes match. * Add `FixedSizeList` handling to: * `requires_nested_struct_cast` * `validate_data_type_compatibility` * Implement recursive casting of nested `Struct` values contained in `FixedSizeList`. * Preserve planner/runtime parity by validating child type compatibility before runtime fallback logic is applied. * Add handling for null-parent `FixedSizeList` entries by masking hidden child values before retrying casts, avoiding failures caused by semantically inaccessible child data. * Refactor list and list-view casting helpers to use Arrow `AsArray` accessors. ## Are these changes tested? Yes. The following tests were added: * `test_cast_fixed_size_list_struct` * `test_validate_fixed_size_list_struct_compatibility` * `test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected` * `test_validate_fixed_size_list_struct_size_mismatch_rejected` * `test_cast_fixed_size_list_struct_all_null` * `test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type` * `test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected` * `test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent` Existing coverage in `test_requires_nested_struct_cast` was also extended to include `FixedSizeList` cases. These tests cover: * Additive nullable nested-field evolution * All-null and partially null list cases * Incompatible nested type changes * Non-nullable field addition rejection * Planner/runtime parity validation ## Are there any user-facing changes? No user-facing changes. This is an internal enhancement to nested schema adaptation and casting behavior for `FixedSizeList<Struct>` types. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. --------- Co-authored-by: Andrew Lamb <[email protected]>

… spark (apache#23766) ## Which issue does this PR close? N/A ## Rationale for this change Hex encoding was implemented six times across the workspace, at two different levels of optimization. Three copies used a fast byte-pair lookup table: - `datafusion/spark/src/function/math/hex.rs` - `datafusion/functions/src/string/to_hex.rs` - `datafusion/functions/src/encoding/inner.rs` (via the `hex` crate) The other three used a slower nibble-at-a-time loop pushing one character at a time: - `datafusion/functions/src/crypto/md5.rs` - `datafusion/spark/src/function/hash/sha1.rs` - `datafusion/spark/src/function/hash/sha2.rs` Beyond the duplication, this split meant the digest functions were paying for a slower encoder than the one already sitting elsewhere in the tree. Consolidating on a single implementation removes the duplication and moves `md5`, `sha1`, and `sha2` onto the fast path. ## What changes are included in this PR? A new `datafusion_common::utils::hex` module holds the only hex encoder in the workspace: ```rust pub enum HexCase { Lower, Upper } pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>); pub fn encode_bytes(bytes: &[u8], case: HexCase) -> String; pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]); pub fn encode_u64(v: u64, case: HexCase, buf: &mut [u8; 16]) -> &[u8]; ``` Four entry points rather than one because the call sites have genuinely different output needs, and forcing them through a single shape would cost an allocation somewhere: `to_hex` writes straight into a `StringArray` values buffer, Spark's `hex` appends to a reused scratch `Vec`, `encode` writes into a pre-sized slice, and the digest functions want an owned `String`. Migrated call sites, all producing bit-identical output: | File | Change | | --- | --- | | `functions/src/string/to_hex.rs` | local table and two write helpers deleted; eight trait impls collapsed to two macros | | `spark/src/function/math/hex.rs` | two nibble tables, two lookup tables, `build_hex_lookup` and `hex_int64` deleted | | `functions/src/crypto/md5.rs` | local table and `hex_encode` deleted | | `spark/src/function/hash/sha1.rs` | local table and inline nibble loop deleted | | `spark/src/function/hash/sha2.rs` | local table and `hex_encode` deleted; eight call sites migrated | | `functions/src/encoding/inner.rs` | both encode sites moved off the `hex` crate | Deliberately left alone: - `hex::decode` / `hex::decode_to_slice` in `encoding/inner.rs`, and Spark's `unhex`. The decode direction has different semantics — Spark's `unhex` left-pads odd-length input, the `hex` crate does not — so unifying it is a separate question. The `hex` dependency stays for those. - `ScalarValue`'s binary `Display` impl in `common/src/scalar/mod.rs`, which writes to a `fmt::Formatter` rather than a byte buffer, and is a cold path. Three details worth a reviewer's attention: - The two ancestors of `encode_u64` disagreed on zero: `to_hex` wrote `'0'` into the caller's buffer, Spark's `hex_int64` returned a `'static` `b"0"` that never touched it. The shared version always writes into the buffer and returns a subslice of it, so the lifetime is uniform. Both callers still produce `"0"`. - Spark's `hex_encode_bytes` guards large binary input with `checked_mul(2)` + `try_reserve`, returning a `DataFusionError` rather than aborting on allocation failure. That guard stays at the call site; `encode_bytes_into` performs no reservation of its own, so behaviour is unchanged. - The encoders are `#[inline(always)]`, not `#[inline]`. They are called once per row from two other crates, and plain `#[inline]` left them out-of-line across the crate boundary. Measured cost of that: +3% on Spark's byte paths and up to +18% on `to_hex`'s i32 path, i.e. the refactor was a net regression on those benchmarks until the attribute changed. ## Are these changes tested? Every migrated function keeps its existing unit and sqllogictest coverage, which is what pins bit-identical output — in particular `spark/hash/sha1.slt` and `sha2.slt` assert concrete digest strings for all four SHA-2 bit lengths across both the scalar and array paths, and `expr.slt` covers `to_hex` and `md5`. New unit tests in `common/src/utils/hex.rs` cover zero, `u64::MAX`, single-nibble values, the odd/even digit-count boundary, two's complement of negative input, empty input, all 256 byte values in both cases, appending into a non-empty buffer, and that a reused scratch buffer never leaks stale digits between calls. Tests cross-check against `format!("{:x}")` rather than restating the implementation. Two Spark tests changed. `test_hex_int64` now drives `hex_encode_int64` instead of the deleted private `hex_int64`, keeping all ten cases including `i64::MIN`, `-1`, and the uppercase expectations. `test_hex_lookup_table_covers_all_bytes` was deleted — it only cross-checked the raw lookup tables, and `encode_bytes_covers_every_byte_value` now does that exhaustively through the public API. A test was added for Spark's lowercase byte path, which previously had no coverage. ### Benchmarks Criterion, `apache/main` @ `eef101769` as baseline. Median of the reported change interval. `datafusion/functions/benches/to_hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `i32_random` | −13.8% | −12.8% | −10.2% | | `i64_random` | −8.8% | −9.5% | −8.1% | | `i64_large_values` | −9.9% | −9.8% | −7.9% | `scalar_i32` −2.3%, `scalar_i64` −1.8%. `datafusion/spark/benches/sha2.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `array_binary_256` | −20.7% | −18.2% | −19.1% | | `array_scalar_binary_256` | −14.6% | −13.2% | −13.1% | `scalar/size=1` −3.3%. `datafusion/spark/benches/hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `hex_int64` | −2.9% | −3.6% | −3.5% | | `hex_int64_dict` | −3.2% | −1.5% | −0.2% | | `hex_utf8` | −0.2% | −0.3% | +0.7% | | `hex_binary` | −0.4% | −0.6% | −0.3% | The `hex_utf8` and `hex_binary` paths already used the byte-pair table before this PR, so they are expected to be flat; they are. `datafusion/functions/benches/crypto.rs`: `md5_array` −4.0%, `md5_scalar` −3.7%. The `sha224` / `sha256` / `sha384` / `sha512` cases in that file range from −0.1% to +2.2%, but they exercise `crypto/basic.rs`, which this PR does not modify and which contains no hex encoding — those numbers are run-to-run variance, not an effect of this change. No number is quoted for Spark `sha1`: it has no benchmark, and its change is the same substitution applied to `md5` and `sha2`. ## Are there any user-facing changes? No behaviour change — all migrated functions produce byte-identical output. `datafusion_common::utils::hex` is new public API on `datafusion-common`. --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Jeffrey Vo <[email protected]>

…t_batch` memory (apache#23873) ## Which issue does this PR close? - related to apache#23716 ## Rationale for this change It adds test coverage for two gaps found while reviewing apache#23716. ## What changes are included in this PR? Tests only, no functional change. - Dictionary inputs - memory usage on retract (make sure memory is released) Note I moved `array_agg` cases out `aggregate.slt` as it is already more than 9k lines long ## Are these changes tested? They are only tests ## Are there any user-facing changes? No. Tests only, no public API changes. --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

…3881) ## Which issue does this PR close? N/A ## Rationale for this change Two Spark functions allocated a `String` per row purely to render a small, bounded amount of text: - `bin` called `format!("{value:b}")` for every row. - `char` called `ch.to_string()` for every row — a heap allocation for a single character. In both cases the output has a known upper bound (64 binary digits for an `i64`, 4 bytes for a UTF-8 character), so the rendering fits in a stack buffer and the result can be appended straight to a pre-sized builder. ## What changes are included in this PR? `math/bin.rs`: - `spark_bin` now writes digits right-aligned into a caller-supplied `[u8; 64]` and returns a `&str` borrowed from it, instead of returning an owned `String`. - The `collect::<StringArray>()` becomes an explicit loop over a `StringBuilder::with_capacity`, sized at 8 digits per row. - Negative values still render as their two's-complement bit pattern, matching `{:b}`. The digit loop is a `loop`, not a `while`, so zero renders as `"0"` rather than the empty string. `string/char.rs`: - `ch.to_string()` becomes `ch.encode_utf8(&mut encoded)` against a `[u8; 4]` hoisted out of the loop. Output is unchanged in both cases. ## Are these changes tested? Existing coverage pins the behaviour. `spark/math/bin.slt` asserts concrete output for the cases the rewrite had to get right: zero, negative values, `i64::MIN` (`-9223372036854775808`), `i64::MAX`, and `-2147483648` / `-32768` widened from narrower integer types. `spark/string/char.slt` covers the negative-input empty string, the null path, and characters on both sides of the ASCII boundary (`char(256)` and above wrap via `% 256`). All 119 `spark/math` and `spark/string` sqllogictest files pass, along with the 258 `datafusion-spark` unit tests. The `bin` benchmark used below, `datafusion/spark/benches/bin.rs`, is added separately in apache#23882 so the baseline can be measured on `main` before this change lands. It covers 1024 and 8192 rows with 20% nulls over two value distributions: small values that render to a handful of digits, and full-range values that render to the maximum 64. `char` already had `datafusion/spark/benches/char.rs` on `main`, so no benchmark change is needed for it. ### Benchmarks Criterion, `apache/main` @ `f1ab86dad` as baseline. Median of the reported change interval. | Benchmark | 1024 | 8192 | | --- | --- | --- | | `bin/small` | −75.7% | −73.7% | | `bin/wide` | −47.8% | −49.8% | `char` (1024 rows): −76.1%. ## Are there any user-facing changes? No. Both functions produce byte-identical output; this is purely an allocation change.

…ts enabled (apache#23848) ## Which issue does this PR close?  - Closes apache#23847 . ## Rationale for this change Fixes a correctness bug, not sure when that config used/desirable. This is another thing I ran into during apache#21585 ## What changes are included in this PR? 1. New SLT test 2. Fix for bug in `datafusion-optimizer` ## Are these changes tested? Existing tests, and a new SLT test verifying the config change doesn't affect the query result and basic unit test. ## Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <[email protected]>

…lter (apache#23901) ## Which issue does this PR close? - Closes apache#23900. ## Rationale for this change `push_down_filter` infers equi-key predicates across a join's ON keys and pushes them to the opposite side. For a null-aware join (the `LeftAnti` join produced by `NOT IN` with a nullable subquery), an outer predicate on the left key like `outer.id > 5` is rewritten to `sub.id > 5` and pushed onto the subquery input. Since the inferred predicate must be null-rejecting to be pushed, this drops the subquery's NULL rows and breaks the three-valued `NOT IN` semantics — a NULL in the subquery key must reach the join so the result is empty. Same class of bug as apache#23848, in a different rule. ## What changes are included in this PR? - Skip predicate inference in `infer_join_predicates` when `join.null_aware` is set (mirrors the apache#23848 guard on `FilterNullJoinKeys`). - A `push_down_filter` unit test asserting no predicate is inferred onto the subquery side of a null-aware `LeftAnti` join. - SLT coverage for the failing query, plus a `prefer_hash_join = false` / multi-partition variant. ## Are these changes tested? Yes — new unit test (verified it fails without the guard) and SLT cases. The full optimizer lib suite passes. ## Are there any user-facing changes? No, aside from the correctness fix.

## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#23311](apache#23311). - Next in stack: apache#23015. - Extracted from apache#19390. ## Rationale for this change For very small `IN` lists, building or probing a hash table can be more work than just comparing the input value with each constant. For example, for `x IN (10, 20, 30)`, the fast path can behave like: ```text x == 10 OR x == 20 OR x == 30 ``` Because the list is tiny, those comparisons are cheap. The implementation stores the constants in a fixed-size array and checks them with a compact comparison chain. “Branchless” here means the comparisons are combined without stopping at the first match. That can be faster for these small fixed-width lists because the CPU gets a predictable sequence of simple operations instead of hash-table setup and probe logic. For primitive values that are not already plain unsigned integers, this PR keeps the logical Arrow type explicit and uses a matching same-width comparison representation only inside the branchless filter. For example, `Float16` uses `UInt16` storage, `Float32` uses `UInt32` storage, and `TimestampNanosecond` uses `UInt64` storage. `Decimal128` and `IntervalMonthDayNano` use their own 16-byte native representation. This preserves bit-pattern equality while relying on Arrow's native primitive compatibility rules: timestamp timezone metadata and Decimal128 precision/scale metadata may differ, while incompatible primitive representations remain rejected. ## What changes are included in this PR? - Adds a const-generic `BranchlessFilter` for small primitive `IN` lists. - Adds thresholds for when this path is used: - up to 16 values for 1-byte types - up to 8 values for 2-byte types - up to 32 values for 4-byte types - up to 16 values for 8-byte types - up to 4 values for 16-byte types - Keeps dispatch concrete and explicit in `strategy.rs`. - Maps each optimized logical type to the comparison representation used by the branchless filter: - `Int8` -> `UInt8` - `Int16`, `Float16` -> `UInt16` - `Int32`, `Float32`, `Date32`, `Time32` -> `UInt32` - `Int64`, `Float64`, `Date64`, `Time64`, `Timestamp`, `Duration` -> `UInt64` - `Decimal128`, `IntervalMonthDayNano` -> their native 16-byte representation - Leaves larger 1-byte and 2-byte lists on the existing bitmap filters. - Leaves larger 4-byte and 8-byte lists on the existing hash/generic paths. - Leaves wider primitive types such as `Decimal256` and unsupported complex types on the generic path. - Keeps the same `IN` / `NOT IN` null behavior as the rest of the stack. - Adds focused coverage for branchless null handling, signed boundary values, slices, Float16/Float32/Float64 bit patterns, compatible timestamp/Decimal128 metadata, incompatible timestamp units, IntervalMonthDayNano values, and same-width wrong-type probe rejection. ## Are these changes tested? Yes. - `cargo fmt --all -- --check` - `cargo test -p datafusion-physical-expr expressions::in_list --lib` - `cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This is an internal performance optimization only. ## Local benchmark snapshot Built and run with `release-nonlto`, filtered to the relevant small primitive-list rows: ```bash cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline> ``` Filters used: `narrow_integer`, `primitive/i32/small_list`, `primitive/i64/small_list`, `f32/small_list`, `timestamp_ns/small_list`, and `interval_month_day_nano/small_list`. Method: directly compared Criterion's raw sample minima (`min(time / iterations)`) from `sample.json`. Lower is better; changes within +/-5% are treated as noise. Compared baselines: [apache#23311](apache#23311) -> [apache#23014](apache#23014) Relevant scope: small primitive-list rows. Summary: 39 relevant rows, 28 faster, 0 slower, 11 within +/-5%. Largest relevant deltas: | Benchmark | Before | After | Change | |---|---:|---:|---:| | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | <details> <summary>Full relevant table (39 rows)</summary> | Benchmark | Before | After | Change | |---|---:|---:|---:| | `narrow_integer/u8/list=4/match=0%` | 3.86 us | 2.79 us | -27.8% (1.38x faster) | | `narrow_integer/u8/list=4/match=50%` | 3.84 us | 2.78 us | -27.7% (1.38x faster) | | `narrow_integer/u8/list=16/match=0%` | 3.88 us | 3.85 us | -0.8% (within noise) | | `narrow_integer/u8/list=16/match=50%` | 3.84 us | 3.86 us | +0.5% (within noise) | | `narrow_integer/i16/list=4/match=0%` | 3.93 us | 3.18 us | -19.1% (1.24x faster) | | `narrow_integer/i16/list=4/match=50%` | 3.92 us | 3.16 us | -19.5% (1.24x faster) | | `narrow_integer/i16/list=64/match=0%` | 3.96 us | 3.82 us | -3.5% (within noise) | | `narrow_integer/i16/list=64/match=50%` | 3.91 us | 3.80 us | -2.9% (within noise) | | `narrow_integer/i16/list=256/match=0%` | 3.90 us | 3.81 us | -2.5% (within noise) | | `narrow_integer/i16/list=256/match=50%` | 3.97 us | 3.81 us | -4.1% (within noise) | | `narrow_integer/f16/list=4/match=0%` | 3.87 us | 3.16 us | -18.5% (1.23x faster) | | `narrow_integer/f16/list=4/match=50%` | 3.94 us | 3.15 us | -20.2% (1.25x faster) | | `narrow_integer/f16/list=64/match=0%` | 3.87 us | 3.84 us | -0.6% (within noise) | | `narrow_integer/f16/list=64/match=50%` | 3.93 us | 3.85 us | -1.9% (within noise) | | `narrow_integer/f16/list=256/match=0%` | 3.90 us | 3.84 us | -1.5% (within noise) | | `narrow_integer/f16/list=256/match=50%` | 3.87 us | 3.91 us | +1.2% (within noise) | | `nulls/narrow_integer/u8/list=16/match=50%/nulls=20%` | 3.92 us | 4.02 us | +2.5% (within noise) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i32/small_list/list=32/match=0%` | 16.34 us | 13.33 us | -18.5% (1.23x faster) | | `primitive/i32/small_list/list=32/match=50%` | 31.17 us | 13.31 us | -57.3% (2.34x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `primitive/i64/small_list/list=16/match=0%` | 16.34 us | 11.93 us | -27.0% (1.37x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `f32/small_list/list=32/match=0%` | 22.05 us | 13.43 us | -39.1% (1.64x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `timestamp_ns/small_list/list=16/match=0%` | 19.73 us | 12.07 us | -38.8% (1.63x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `interval_month_day_nano/small_list/list=4/match=0%` | 20.12 us | 13.20 us | -34.4% (1.52x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | </details> --------- Co-authored-by: Andrew Lamb <[email protected]>

## Which issue does this PR close? - Closes apache#23770 - Part of #15914 ## Rationale for this change Spark provides [`hypot(expr1, expr2)`](https://spark.apache.org/docs/latest/api/sql/#hypot), which returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or underflow. It was not yet implemented in `datafusion-spark` — only an auto-generated test stub existed at `spark/math/hypot.slt` with its query commented out. ## What changes are included in this PR? - Add `SparkHypot` (implementing `ScalarUDFImpl`) in `datafusion/spark/src/function/math/hypot.rs`, backed by Rust's `f64::hypot` — the same overflow-safe algorithm as Java/Spark's `Math.hypot`. - Register it in `datafusion/spark/src/function/math/mod.rs`. - Enable the `hypot.slt` sqllogictest. The signature is `exact(Float64, Float64) -> Float64`, following the `datafusion-spark` convention of only accepting types Spark supports. Computation uses the Arrow `binary` kernel so NULL in either argument propagates to a NULL result, matching Spark. ## Are these changes tested? Yes — `datafusion/sqllogictest/test_files/spark/math/hypot.slt` covers: - scalar Pythagorean triples (`hypot(3, 4)` → 5, `hypot(5, 12)` → 13), - double inputs, - NULL propagation when either argument is NULL, - the array path (including a NULL row), - overflow-safety: `hypot(3e200, 4e200)` stays finite, whereas a naive `sqrt(a^2 + b^2)` would overflow to `Infinity`. ## Are there any user-facing changes? Yes — adds the Spark-compatible `hypot` scalar function to `datafusion-spark`. No breaking changes to public APIs.

## Which issue does this PR close?  - Part of apache#23393 . ## Rationale for this change `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` only reported the stack size of their `ScalarValue` field plus its heap payload, ignoring the memory held by the underlying `MovingMin` / `MovingMax` sliding-window buffers. For windowed `MIN`/`MAX` over string or list data, the two per-element stacks can hold megabytes of `ScalarValue` payload that the memory pool was never told about, understating accumulator memory usage. ## What changes are included in this PR? - Add a private `heap_size(elem_heap)` method to `MovingMin<T>` and `MovingMax<T>` that reports the two stack buffers' capacity in bytes plus each stored element's heap payload. - Factor the shared implementation into a `moving_stacks_heap_size` free helper so the two types stay in sync. - Include the buffer bytes in `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` via the new method. ## Are these changes tested? Yes. Two new unit tests in `datafusion/functions-aggregate/src/min_max.rs`: - `moving_min_max_heap_size_i32` — fixed-width `T`, verifies buffer-only accounting with and without pushed elements. - `moving_min_max_heap_size_counts_elems` — `T = String`, verifies each of the two slots in a `(T, T)` pair contributes independently to the heap payload (mirroring the two independent `Clone`s made by `push`).

## Which issue does this PR close? - Related to apache#21882 (comment). ## Rationale for this change PR apache#21882 adds custom spill file support. Having an example showing how a downstream application can use this new API will help make sure the API is good enough for our needs ## What changes are included in this PR? This PR adds a small example showing how users can back spill files with an `ObjectStore`, using a local object store for a runnable example while keeping the implementation applicable to remote stores. ## Are these changes tested? Yes by CI ## Are there any user-facing changes? Yes. This adds a new example for configuring ObjectStore-backed spill files.

## Which issue does this PR close?  - Closes #N/A. ## Rationale for this change  Since DataFusion doesn't typically guartantee wire format compatibility, I remove the backward compatiblity shim that introduced in PR apache#23189 FYI: apache#23189 (comment) ## What changes are included in this PR?  DataFusion does not guarantee serialized plans across versions. Keeping `partitioned_by_file_group` therefore leaves a dead schema field and decoder path after `output_partitioning` became the source of truth. Reserve the old field number and name to prevent future reuse. ## Are these changes tested?  Yes ## Are there any user-facing changes?   Yes Signed-off-by: Jiawei Zhao <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

…llable build key (apache#23173) ## Which issue does this close? - Closes apache#23126. ## Rationale for this change `x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build = outer `x`, probe = subquery). Join dynamic filter pushdown pushes a bounds + membership filter, built from the build keys, onto the probe scan. That filter can prune every probe row. A null-aware `LeftAnti` reads an empty probe as a genuinely-empty subquery, so it emits build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is UNKNOWN, not TRUE. The result is scan-dependent, so it's a silent correctness bug. A `VALUES` scan ignores the pushed filter and stays correct; a parquet scan applies it and is wrong. apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side NULL. ## What changes are included in this PR? Skip join dynamic filter pushdown for a null-aware anti join when the build key can be NULL. The build-side NULL emission depends on whether the probe is truly empty, which the pushed filter can change by emptying it. A NOT NULL build key has no such NULL, so it keeps the pushdown. The check is static: a schema-nullable build key disables the pushdown even when the data contains no NULLs. A runtime alternative (keep the pushdown and neutralize the filter only when the build actually holds a NULL key) would restore the optimization for those cases. I'd leave that as a follow-up. ## Are these changes tested? Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side NULL, a non-matching parquet probe) and asserts the single correct row. Without the change it returns the extra NULL. In addition, unit tests pin both directions of the guard: a nullable build key rejects the pushdown and a NOT NULL build key keeps it. ## Are there any user-facing changes? `NOT IN` over a parquet (or otherwise prunable) scan with a nullable outer key now returns correct results. Such joins lose the dynamic filter pushdown.

…cimal) (apache#23631) ## Which issue does this PR close?  N/A ## Rationale for this change  I noticed there were some subtle errors with how decimals were handled in scalar values, and also opportunity to remove power calls in favour of precomputed constant tables. Also filling out some other missing support. ## What changes are included in this PR?  I recommend looking at the commits as they are self contained with detailed messages for each. ## Are these changes tested?  Yes ## Are there any user-facing changes?  No

…lter pushdown enabled (apache#23638) ## Which issue does this PR close? - Closes apache#23531. ## Rationale for this change `input_file_name()` is `FileSource` dependent like `file_row_index()`, and therefore shouldn't be pushed down into a filter. ## What changes are included in this PR? `PushdownChecker` now handles both UDFs consistently. If we keep adding this sort of metadata functions, we might want a better API to detect them, but for now I think this is a reasonable approach that isn't very invasive. ## Are these changes tested? Additional SLT test that verifies that both function behave correctly with pushdown either enabled or disabled. ## Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <[email protected]>

…xpr::check_bigger_cast (apache#23808) (apache#23809) ## Which issue does this PR close? - Closes apache#23808. ## Rationale for this change ref. apache#23807 `CastExpr::check_bigger_cast` is used to determine whether a cast is a widening (order-preserving) conversion. Currently, it classifies `Int32 -> Float32`, `UInt32 -> Float32`, `Int64 -> Float64`, and `UInt64 -> Float64` as widening casts. However, integer-to-float conversions for 32-bit and 64-bit integers lose precision when values exceed the mantissa bit limit (24 bits for `Float32`, 53 bits for `Float64`). For example: `16_777_216_i32 as f32 == 16_777_217_i32 as f32` (both yield 16777216.0f32). Because distinct integer inputs can collapse to the same float output, these casts are not strictly 1-to-1 (injective) and can break suffix sort key ordering in multi-column ordering analysis (e.g. `[CAST(a AS Float32), b]`). ## What changes are included in this PR? - Updated `CastExpr::check_bigger_cast` to exclude precision-losing integer-to-float conversions (`Int32/UInt32 -> Float32` and `Int64/UInt64 -> Float64`). - Added unit test `test_check_bigger_cast_precision_loss` to verify precision-losing casts return `false` while exact conversions (`Int16 -> Float32`, `Int32 -> Float64`, etc.) continue to return `true`. ## Are these changes tested? Yes, new unit test `test_check_bigger_cast_precision_loss` in `cast.rs`. ## Are there any user-facing changes? No breaking API changes. Internal optimizer behavior fix.

…ossible (apache#23702) ## Which issue does this PR close? N/A ## Rationale for this change `SortPreservingMergeStream` is a little complex, so add some guiding comments and make it as textbook-like as possible ## What changes are included in this PR? Added comments, reorder code While this was done this also fixed couple of bugs due to how it work: 1. leftover drain was not counted in the `elapsed_compute` 2. `limit(0)` returns 0 rows and not 1 ## Are these changes tested? Existing tests ## Are there any user-facing changes? Not API ones. `limit(0)` now returns 0 rows

## Which issue does this PR close? - Closes apache#21428. ## Rationale for this change This PR adds `BuildHasher`-based variants for `hash_utils` so callers can compute row hashes with a caller-provided hash builder instead of always using DataFusion's default `RandomState`. The main constraint is performance: `with_hashes` is a hot path, especially for string, dictionary, and nested array hashing. A previous version in apache#21429 caused measurable regressions in the default `RandomState` path, for example `large_utf8: single, no nulls` regressed from roughly `26.7us` to `36.3us`, and `large_utf8: multiple, no nulls` from roughly `112us` to `127us`. This version keeps the default path performance-oriented by avoiding a fully generic `BuildHasher` rewrite of the existing hot loops. ## What changes are included in this PR? This PR adds: - `with_hashes_with_hasher` - `create_hashes_with_hasher` - custom-hasher implementations for primitive, string, binary, byte-view, dictionary, and nested arrays - tests covering custom hashers, multi-column hashing, and dictionary equivalence The implementation intentionally uses a hybrid design: - Default `RandomState` leaf hot paths remain specialized. - Custom `BuildHasher` leaf paths live separately in `hash_utils/build_hasher.rs`. - Nested/structural logic is shared through an internal child-hashing adapter, so struct/list/map/union/run/dictionary behavior does not need to be broadly duplicated. The trade-off is that there is still some duplication for primitive/string/binary leaf loops. That duplication is intentional: those are the hottest loops, and keeping them separate prevents the existing `RandomState` path from becoming generic over `BuildHasher` or being perturbed by the custom-hasher implementation. ## Are these changes tested? Yes. --------- Co-authored-by: Dmitrii Blaginin <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

…t whether they keep the same ordering of the input (apache#23807) ## Which issue does this PR close? - Closes apache#23798 Related to: - apache#16217 ## Rationale for this change To be able to keep the same sorting order allowing for more optimizations Now comet own `cast` implementation can be recognized as not modifying sort order in the same cases that datafusion cast does. ## What changes are included in this PR? added `strictly_order_preserving` property to `ExprProperties` + varius other places and replaced the hard coded logic for cast (`substitute_cast_ordering`) about keeping input order with more generic approach that now any expression can implement and have the same advantage of sort elimination also marked from_unixtime as keeping ordering to show a case of this optimization ## Are these changes tested? yes ## Are there any user-facing changes? yes, breaking change, added `strictly_order_preserving` property to `ExprProperties` and to `FFI_ExprProperties` this property means that given expression `f` and 2 values from the input column `a` and `b` the following variants are kept: 1. `a.cmp(b) == f(a).cmp(f(b))` 2. nulls maps to nulls Example of satisfying expression: `cast(col_a as BIGINT)` where `col_a` is `INT` it is keeping the properties Example of not satisfying: `floor` - floor can not `array_repeat(my_col, 2)` which might look like at first glance as keeping the property as well but in fact it does not. the reason is that `array_repeat(null, 2)` will output list of 2 nulls which breaks the 2nd property that nulls must be kept as nulls how to migrate: Option 1 - keeping the old behavior (safest but least performant) set `strictly_order_preserving` to false Option 2 - Using the new optimization that this opens up: set `strictly_order_preserving` only if the expression keep both variants

## Which issue does this PR close? Part of apache#22330. ## Rationale for this change `ForeignScalarUDF` inherits the default `preserves_lex_ordering`, so producer overrides are lost across the FFI boundary. ## What changes are included in this PR? - Forward `preserves_lex_ordering` through `FFI_ScalarUDF`. - Reuse the existing placement UDF for unit and dynamic-library coverage. ## Are these changes tested? - `cargo test -p datafusion-ffi --features integration-tests` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? The FFI ABI changes. Foreign libraries must rebuild against the new DataFusion version. --------- Signed-off-by: Amogh Ramesh <[email protected]>

…tonic deques (apache#23826) (apache#23827) ## Which issue does this PR close? - Closes apache#23826 ### What changes are included in this PR? This PR optimizes sliding window `MIN`/`MAX` aggregate functions using a **Sequence-Numbered Monotonic Deque** instead of a Two-Stack Queue. **Revised Design:** - We store `(sequence_number, value)` pairs in a single `VecDeque`, kept in strictly monotonic order. - **`push(val)`**: Evicts dominated elements from the back, then pushes the new value with the current `push_seq` and increments `push_seq`. - **`pop()`**: Increments `pop_seq`. If the front elements sequence number equals the old `pop_seq`, it is expired and popped. - **Benefits**: No secondary FIFO queue (lower memory) and no `clone()` overhead. ### Are these changes tested? Yes, existing tests pass. Added tests for empty-window `pop()` and duplicate-heavy scenarios. ### Are there any user-facing changes? **API Change**: `MovingMin` and `MovingMax` were changed to `pub(crate)` visibility, and their `pop()` methods now return `()` instead of returning a value. Performance is significantly improved (2x-3.5x throughput). --------- Co-authored-by: Pavan <[email protected]>

## Which issue does this PR close? NA ## Rationale for this change Adds [datapress](https://docs.datap-rs.org) to the list of known users in the documentation. ## What changes are included in this PR? NA ## Are these changes tested? NA ## Are there any user-facing changes? NA

## Which issue does this PR close? - Closes apache#11748 ## Rationale for this change `EliminateGroupByConstant` removes GROUP BY expressions that are constants. If all of the GROUP BY expressions are constants, eliminating all of them results in converting a grouped aggregate into an ungrouped (global) aggregate. This changes the semantics of the query: a grouped aggregate query on an empty input returns zero rows, whereas an ungrouped aggregate query produces a single row. ## What changes are included in this PR? `EliminateGroupByConstant` now declines to eliminate constant GROUP BY expressions, if doing so would result in removing all of the grouping expressions. ## Are these changes tested? Yes. Sqllogictest reproducer for the original issue, updated unit test and optimizer SLT expectations. ## Are there any user-facing changes? Queries with all-constant GROUP BY may now return fewer (correct) rows.

…pache#23874) ## Which issue does this PR close? - Closes apache#23872 ## Rationale for this change `SlidingMinAccumulator::update_batch` skipped NULL values. This meant that if all the non-NULL values in a window frame were retracted, the window frame would not be empty but the `MovingMin` data structure by the `SlidingMinAccumulator` would not contain any values. This resulted in incorrectly returning a stale non-NULL value for a sliding `min()` over a window frame consisting of only NULL values. Along the way, optimize the min and max sliding window accumulators to make them both more efficient and more symmetric with one another. In the original coding, `SlidingMaxAccumulator` included NULL values but `SlidingMinAccumulator` omitted them, in part because omitting NULLs made the original `retract_batch` implementation more expensive. This PR optimizes `retract_batch`, so we can now use the same scheme for both the min and max sliding accumulators: * Omit NULLs on `update_batch` (this improves on the prior behavior of `max`) * Efficiently account for NULLs in `retract_batch` (this improves on the prior behavior of `min`) * Ensure correct results for all-NULL window frames (this fixes the prior bug in `min`). ## What changes are included in this PR? * Fix bug in `min()` over all-NULL window frames * Optimize `SlidingMinAccumulator::retract_batch` (avoid materializing values just to count NULLs) * Optimize `SlidingMinAccumulator::update_batch` (omit NULLs), also improving symmetry with `min` * Optimize both accumulators to stop caching the current `min` / `max`; this saves a few clones, but perhaps more importantly it is simpler and avoids the risk of inconsistency between the cached value and the underlying `MovingMin` / `MovingMax` data structure ## Are these changes tested? Yes, new tests added. ## Are there any user-facing changes? No, aside from the bug fix.

## Which issue does this PR close? - N/A ## Rationale for this change ``` $ cargo test -p datafusion-functions-aggregate [...] warning: associated function `from_parts` is never used --> datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs:464:8 [...] ``` Fix this by appropriately gating the compilation of `from_parts`. ## What changes are included in this PR? * Squelch unused code warning ## Are these changes tested? Yes. ## Are there any user-facing changes? No.

…w retract (apache#23913) ## Which issue does this PR close? - Closes apache#23912. ## Rationale for this change `DistinctPercentileContAccumulator` reused the shared set-based `GenericDistinctBuffer`, which doesn't fit it: (1) the buffer asserts a single input column but `percentile_cont` passes two (value + percentile), so every `percentile_cont(DISTINCT ...)` panicked; (2) the buffer is a plain `HashSet` with no multiplicity, so sliding-window `retract_batch` dropped a value while duplicates were still in the frame. ## What changes are included in this PR? - Replace the shared buffer in this accumulator with a per-accumulator `HashMap<Hashable, usize>` count map: `update_batch` reads only the value column and increments; `retract_batch` decrements and removes a key only at zero; `state`/`merge_batch` keep the same List state shape. Other `GenericDistinctBuffer` users are untouched. - Regression tests in `aggregate.slt` for the plain distinct query and the sliding-window duplicate-retract case. ## Are these changes tested? Yes — new regression tests; the full `aggregate.slt` suite passes. ## Are there any user-facing changes? `percentile_cont(DISTINCT ...)` now works instead of panicking, and returns correct results in sliding windows.

## Which issue does this PR close? - Closes apache#23717. ## Rationale for this change Spark and Java format negative numeric values with parentheses when the `(` flag is present. The decimal formatting path always emitted a minus sign because it ignored `negative_in_parentheses`, while the floating-point path already handled the flag. This made `format_string` inconsistent across numeric input types. ## What changes are included in this PR? - Add a closing suffix when a negative decimal uses parentheses formatting. - Include the suffix when calculating width, left alignment, and zero padding. - Add regression coverage for grouped negative decimals with and without an explicit width. ## Are these changes tested? Yes. The following checks pass locally: - `cargo fmt --all -- --check` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-spark --lib` - The full workspace test command from the contributor guide with `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption` enabled ## Are there any user-facing changes? Yes. Spark-compatible formatting of negative decimal values now honors the parentheses flag. There are no public API or breaking changes.

## Which issue does this PR close?  - Closes #. ## Rationale for this change  Precedes apache#23720. While displaying a plan with metrics, allow filtering them by name, not just by category or type. This is very useful when writing snapshot tests where only a specific metrics needs to be asserted, without other unrelated metrics polluting the snapshot assertion. Regardless of what happens with apache#23720, I think this PR is still worth it, as it allows creating some really nice `insta` tests asserting runtime properties reliably by just cherry picking the runtime metrics relevant for that specific test. This is relevant not only within DataFusion codebase, but also for other people's codebases using DataFusion and relying on `insta` for snapshot testing, See an example of this here: https://github.com/apache/datafusion/pull/23720/changes#diff-8281405e117428077c19c07afbbe59fed303a3460096e958f761d34f0899b618 ## What changes are included in this PR?  While displaying metrics, allows filtering them by name ## Are these changes tested?  Yes, by a new small test. ## Are there any user-facing changes?   People using the metrics display API will be able to filter metrics by name.

## Which issue does this PR close? N/A ## Rationale for this change I saw that `empty` implementation is inefficient. I wrote review comments for more why inefficient. there are some optimizations that do not require benchmark since they are obvious once you understand, this is one of them ## What changes are included in this PR? rewrote the function to be fast ## Are these changes tested? existing tests ## Are there any user-facing changes? no

@ref

Bumps the codeql-actions group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.1 to 4.37.3 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/init's releases</a>.</em></p> <blockquote> <h2>v4.37.3</h2> <p>No user facing changes.</p> <h2>v4.37.2</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/init's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.37.2 - 21 Jul 2026</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> <h2>4.37.1 - 16 Jul 2026</h2> <ul> <li><em>Upcoming breaking change</em>: Add a deprecation warning for customers using CodeQL version 2.20.6 and earlier. These versions of CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise Server 3.16, and will be unsupported by the next minor release of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li> </ul> <h2>4.37.0 - 08 Jul 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li> <li>In addition to the existing input format, the <code>config-file</code> input for the <code>codeql-action/init</code> step will soon support a new <code>[owner/]repo[@ref][:path]</code> format. All components except the repository name are optional. If omitted, <code>owner</code> defaults to the same owner as the repository the analysis is running for, <code>ref</code> to <code>main</code>, and <code>path</code> to <code>.github/codeql-action.yaml</code>. Support for this format ships in this version of the CodeQL Action, but will only be enabled over the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li> </ul> <h2>4.36.3 - 01 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.36.2 - 04 Jun 2026</h2> <ul> <li>Cache CodeQL CLI version information across Actions steps. <a href="https://redirect.github.com/github/codeql-action/pull/3943">#3943</a></li> <li>Reduce requests while waiting for analysis processing by using exponential backoff when polling SARIF processing status. <a href="https://redirect.github.com/github/codeql-action/pull/3937">#3937</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.6">2.25.6</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3948">#3948</a></li> </ul> <h2>4.36.1 - 02 Jun 2026</h2> <p>No user facing changes.</p> <h2>4.36.0 - 22 May 2026</h2> <ul> <li><em>Breaking change</em>: Bump the minimum required CodeQL bundle version to 2.19.4. <a href="https://redirect.github.com/github/codeql-action/pull/3894">#3894</a></li> <li>Add support for SHA-256 Git object IDs. <a href="https://redirect.github.com/github/codeql-action/pull/3893">#3893</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.5">2.25.5</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3926">#3926</a></li> </ul> <h2>4.35.5 - 15 May 2026</h2> <ul> <li>We have improved how the JavaScript bundles for the CodeQL Action are generated to avoid duplication across bundles and reduce the size of the repository by around 70%. This should have no effect on the runtime behaviour of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3899">#3899</a></li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81"><code>e4fba86</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4031">#4031</a> from github/update-v4.37.3-72f6a9da0</li> <li><a href="https://github.com/github/codeql-action/commit/fb50ab5d62a274adf3ef3e22cfe750ae87a0ede7"><code>fb50ab5</code></a> Update changelog for v4.37.3</li> <li><a href="https://github.com/github/codeql-action/commit/72f6a9da0def52d9193d6a758f0378b65091f8d1"><code>72f6a9d</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4030">#4030</a> from github/mbg/fix/no-proxy</li> <li><a href="https://github.com/github/codeql-action/commit/3b5ee58597653d9cc6785f3f1277f796d81f3646"><code>3b5ee58</code></a> Use default <code>request</code> options instead of <code>undefined</code></li> <li><a href="https://github.com/github/codeql-action/commit/bfb6be4b5ecd3650f02f530571453e8c64ef0778"><code>bfb6be4</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4028">#4028</a> from github/mergeback/v4.37.2-to-main-e0647621</li> <li><a href="https://github.com/github/codeql-action/commit/526ab84f9858816d9cf5f7b9df4dd5e2235f0eba"><code>526ab84</code></a> Rebuild</li> <li><a href="https://github.com/github/codeql-action/commit/d6217b9b8c14166e4851db94c11155d03bd13c07"><code>d6217b9</code></a> Update changelog and version after v4.37.2</li> <li><a href="https://github.com/github/codeql-action/commit/e0647621c2984b5ed2f768cb892365bf2a616ad1"><code>e064762</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4027">#4027</a> from github/update-v4.37.2-385bcdc5a</li> <li><a href="https://github.com/github/codeql-action/commit/e0faed839190caa67a5cd42f1cc16246028ca3df"><code>e0faed8</code></a> Add a couple of change notes</li> <li><a href="https://github.com/github/codeql-action/commit/73aad0eaa9df172668665a150d17b8bc5a650c20"><code>73aad0e</code></a> Update changelog for v4.37.2</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/7188fc363630916deb702c7fdcf4e481b751f97a...e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81">compare view</a></li> </ul> </details> <br /> Updates `github/codeql-action/analyze` from 4.37.1 to 4.37.3 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/analyze's releases</a>.</em></p> <blockquote> <h2>v4.37.3</h2> <p>No user facing changes.</p> <h2>v4.37.2</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/analyze's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.37.2 - 21 Jul 2026</h2> <ul> <li>The new address format for the <code>config-file</code> input that was introduced in CodeQL Action 4.37.0 is now enabled by default. In addition to the format described there, the <code>remote=</code> prefix can now be used to explicitly indicate that the input refers to a remote file. All previous input formats continue to be accepted as well. <a href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li> <li>The CodeQL Action can now make use of <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured private registries</a> in Default Setup to retrieve CodeQL configuration files from remote repositories that require authentication. This will allow customers to store their CodeQL configuration in a single repository that can then be referenced by Default Setup workflows in other repositories. We expect to roll this and other, related changes out to everyone in July. <a href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li> </ul> <h2>4.37.1 - 16 Jul 2026</h2> <ul> <li><em>Upcoming breaking change</em>: Add a deprecation warning for customers using CodeQL version 2.20.6 and earlier. These versions of CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise Server 3.16, and will be unsupported by the next minor release of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li> </ul> <h2>4.37.0 - 08 Jul 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li> <li>In addition to the existing input format, the <code>config-file</code> input for the <code>codeql-action/init</code> step will soon support a new <code>[owner/]repo[@ref][:path]</code> format. All components except the repository name are optional. If omitted, <code>owner</code> defaults to the same owner as the repository the analysis is running for, <code>ref</code> to <code>main</code>, and <code>path</code> to <code>.github/codeql-action.yaml</code>. Support for this format ships in this version of the CodeQL Action, but will only be enabled over the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li> </ul> <h2>4.36.3 - 01 Jul 2026</h2> <p>No user facing changes.</p> <h2>4.36.2 - 04 Jun 2026</h2> <ul> <li>Cache CodeQL CLI version information across Actions steps. <a href="https://redirect.github.com/github/codeql-action/pull/3943">#3943</a></li> <li>Reduce requests while waiting for analysis processing by using exponential backoff when polling SARIF processing status. <a href="https://redirect.github.com/github/codeql-action/pull/3937">#3937</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.6">2.25.6</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3948">#3948</a></li> </ul> <h2>4.36.1 - 02 Jun 2026</h2> <p>No user facing changes.</p> <h2>4.36.0 - 22 May 2026</h2> <ul> <li><em>Breaking change</em>: Bump the minimum required CodeQL bundle version to 2.19.4. <a href="https://redirect.github.com/github/codeql-action/pull/3894">#3894</a></li> <li>Add support for SHA-256 Git object IDs. <a href="https://redirect.github.com/github/codeql-action/pull/3893">#3893</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.5">2.25.5</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3926">#3926</a></li> </ul> <h2>4.35.5 - 15 May 2026</h2> <ul> <li>We have improved how the JavaScript bundles for the CodeQL Action are generated to avoid duplication across bundles and reduce the size of the repository by around 70%. This should have no effect on the runtime behaviour of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3899">#3899</a></li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81"><code>e4fba86</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4031">#4031</a> from github/update-v4.37.3-72f6a9da0</li> <li><a href="https://github.com/github/codeql-action/commit/fb50ab5d62a274adf3ef3e22cfe750ae87a0ede7"><code>fb50ab5</code></a> Update changelog for v4.37.3</li> <li><a href="https://github.com/github/codeql-action/commit/72f6a9da0def52d9193d6a758f0378b65091f8d1"><code>72f6a9d</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4030">#4030</a> from github/mbg/fix/no-proxy</li> <li><a href="https://github.com/github/codeql-action/commit/3b5ee58597653d9cc6785f3f1277f796d81f3646"><code>3b5ee58</code></a> Use default <code>request</code> options instead of <code>undefined</code></li> <li><a href="https://github.com/github/codeql-action/commit/bfb6be4b5ecd3650f02f530571453e8c64ef0778"><code>bfb6be4</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4028">#4028</a> from github/mergeback/v4.37.2-to-main-e0647621</li> <li><a href="https://github.com/github/codeql-action/commit/526ab84f9858816d9cf5f7b9df4dd5e2235f0eba"><code>526ab84</code></a> Rebuild</li> <li><a href="https://github.com/github/codeql-action/commit/d6217b9b8c14166e4851db94c11155d03bd13c07"><code>d6217b9</code></a> Update changelog and version after v4.37.2</li> <li><a href="https://github.com/github/codeql-action/commit/e0647621c2984b5ed2f768cb892365bf2a616ad1"><code>e064762</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4027">#4027</a> from github/update-v4.37.2-385bcdc5a</li> <li><a href="https://github.com/github/codeql-action/commit/e0faed839190caa67a5cd42f1cc16246028ca3df"><code>e0faed8</code></a> Add a couple of change notes</li> <li><a href="https://github.com/github/codeql-action/commit/73aad0eaa9df172668665a150d17b8bc5a650c20"><code>73aad0e</code></a> Update changelog for v4.37.2</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/7188fc363630916deb702c7fdcf4e481b751f97a...e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…e#23941) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.84.0 to 2.85.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's releases</a>.</em></p> <blockquote> <h2>2.85.2</h2> <ul> <li> <p>Update <code>prek@latest</code> to 0.4.11.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.7.13.</p> </li> <li> <p>Update <code>kingfisher@latest</code> to 1.109.0.</p> </li> </ul> <h2>2.85.1</h2> <ul> <li> <p>Update <code>vacuum@latest</code> to 0.30.0.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.11.32.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.7.12.</p> </li> <li> <p>Update <code>cyclonedx@latest</code> to 0.33.1.</p> </li> <li> <p>Update <code>cargo-neat@latest</code> to 0.5.2.</p> </li> </ul> <h2>2.85.0</h2> <ul> <li> <p>Support <code>wild</code> (alias: <code>wild-linker</code>). (<a href="https://redirect.github.com/taiki-e/install-action/pull/1949">#1949</a>)</p> </li> <li> <p>Support <code>bpf-linker</code>. (<a href="https://redirect.github.com/taiki-e/install-action/pull/1950">#1950</a>)</p> </li> <li> <p>Support <code>rafn</code>. (<a href="https://redirect.github.com/taiki-e/install-action/pull/1935">#1935</a>, thanks <a href="https://github.com/DarkWanderer"><code>@DarkWanderer</code></a>)</p> </li> <li> <p>Update <code>cargo-neat@latest</code> to 0.5.1.</p> </li> <li> <p>Update <code>zizmor@latest</code> to 1.28.0.</p> </li> <li> <p>Update <code>wasmtime@latest</code> to 47.0.2.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.11.31.</p> </li> <li> <p>Update <code>syft@latest</code> to 1.49.0.</p> </li> </ul> <h2>2.84.1</h2> <ul> <li> <p>Update <code>wasmtime@latest</code> to 47.0.1.</p> </li> <li> <p>Update <code>wasm-tools@latest</code> to 1.254.0.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.11.30.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.7.11.</p> </li> <li> <p>Update <code>cargo-neat@latest</code> to 0.5.0.</p> </li> <li> <p>Update <code>cargo-crap@latest</code> to 0.3.1.</p> </li> <li> <p>Update <code>biome@latest</code> to 2.5.5.</p> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <p>All notable changes to this project will be documented in this file.</p> <p>This project adheres to <a href="https://semver.org">Semantic Versioning</a>.</p>  <h2>[Unreleased]</h2> <h2>[2.85.2] - 2026-07-26</h2> <ul> <li> <p>Update <code>prek@latest</code> to 0.4.11.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.7.13.</p> </li> <li> <p>Update <code>kingfisher@latest</code> to 1.109.0.</p> </li> </ul> <h2>[2.85.1] - 2026-07-25</h2> <ul> <li> <p>Update <code>vacuum@latest</code> to 0.30.0.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.11.32.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.7.12.</p> </li> <li> <p>Update <code>cyclonedx@latest</code> to 0.33.1.</p> </li> <li> <p>Update <code>cargo-neat@latest</code> to 0.5.2.</p> </li> </ul> <h2>[2.85.0] - 2026-07-23</h2> <ul> <li> <p>Support <code>wild</code> (alias: <code>wild-linker</code>). (<a href="https://redirect.github.com/taiki-e/install-action/pull/1949">#1949</a>)</p> </li> <li> <p>Support <code>bpf-linker</code>. (<a href="https://redirect.github.com/taiki-e/install-action/pull/1950">#1950</a>)</p> </li> <li> <p>Support <code>rafn</code>. (<a href="https://redirect.github.com/taiki-e/install-action/pull/1935">#1935</a>, thanks <a href="https://github.com/DarkWanderer"><code>@DarkWanderer</code></a>)</p> </li> <li> <p>Update <code>cargo-neat@latest</code> to 0.5.1.</p> </li> <li> <p>Update <code>zizmor@latest</code> to 1.28.0.</p> </li> <li> <p>Update <code>wasmtime@latest</code> to 47.0.2.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.11.31.</p> </li> <li> <p>Update <code>syft@latest</code> to 1.49.0.</p> </li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/taiki-e/install-action/commit/41049aa56687c35e0afa74eed4f09cec4f9afabf"><code>41049aa</code></a> Release 2.85.2</li> <li><a href="https://github.com/taiki-e/install-action/commit/dfcf36552b1b743e910b9b9b6b114a08e56a1e5e"><code>dfcf365</code></a> Update <code>prek@latest</code> to 0.4.11</li> <li><a href="https://github.com/taiki-e/install-action/commit/eea03ccfa855b965a301aa52424915fddc32f855"><code>eea03cc</code></a> Update <code>mise@latest</code> to 2026.7.13</li> <li><a href="https://github.com/taiki-e/install-action/commit/81ca2feb84748ed3fa514707ee1004f4f3ad715a"><code>81ca2fe</code></a> Update martin manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/cc90ed04bc1a258ea90a83789f24b759a77c9671"><code>cc90ed0</code></a> Update <code>kingfisher@latest</code> to 1.109.0</li> <li><a href="https://github.com/taiki-e/install-action/commit/55639a3362f508fda5154d3451072205f5c32ca6"><code>55639a3</code></a> Update cargo-shear manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/3d7d7cd5ac7f994c1892ae0c06165095b9139094"><code>3d7d7cd</code></a> Release 2.85.1</li> <li><a href="https://github.com/taiki-e/install-action/commit/d09ccb4fe2105ac9ead6376f7a423ff88defeb57"><code>d09ccb4</code></a> Update <code>vacuum@latest</code> to 0.30.0</li> <li><a href="https://github.com/taiki-e/install-action/commit/ac43dee1a92e482c0e244bdb0bb126908159e245"><code>ac43dee</code></a> Update <code>uv@latest</code> to 0.11.32</li> <li><a href="https://github.com/taiki-e/install-action/commit/49b16979f38f30e44b1f68af9115be9a6ffc5215"><code>49b1697</code></a> Update prek manifest</li> <li>Additional commits viewable in <a href="https://github.com/taiki-e/install-action/compare/a6b2e2dcd845ddd7f509ce4f3ed3d922b80cc5d9...41049aa56687c35e0afa74eed4f09cec4f9afabf">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=taiki-e/install-action&package-manager=github_actions&previous-version=2.84.0&new-version=2.85.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Copilot

Copilot review overview

🔵 Needs a closer look

The 900-commit, 1,720-file merge spans core execution, optimizer, API, tooling, and benchmark behavior and requires final human validation.

Review effort: Balanced
Findings: None

What changed in this PR

Merges DataFusion 55.1.0 into spiceai-55 while retaining Spice-specific correctness and compatibility patches.

Changes:

Integrates upstream optimizer, execution, API, dependency, and benchmark updates.
Fixes join sum statistics and dialect-aware Date32 unparsing.
Preserves Spark feature isolation and Rust 1.94 compatibility.

File	Description
`datafusion/**`	Upstream engine changes and Spice fixes
`benchmarks/**`	New benchmark framework and suites
`docs/**`	55.1 documentation and upgrade guidance
`.github/**`	CI and dependency automation updates
Root configuration files	Toolchain, licensing, and lint updates

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…o spiceai-59 (#244)

Copilot

Copilot review overview

🔵 Needs a closer look

The 1,720-file, 901-commit merge requires human validation, and its documented Arrow revision does not match the committed pin.

Review effort: Balanced
Findings: 1

Open (1)

Align documented Arrow revision with the committed dependency pin · New

…into spiceai-55

…nowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code).

* build: pin arrow-rs at 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 * build: pin DataFusion at 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55

…into spiceai-55 (#53)

* build: pin arrow-rs at 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 * build: pin DataFusion at 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55

…into spiceai-55 (#53)

* build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing spiceai#237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for spiceai#237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (spiceai#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (spiceai#70, spiceai#71, spiceai#72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (spiceai#82, spiceai#53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, spiceai#54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, spiceai#15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, spiceai#16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, spiceai#7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, spiceai#50), datafusion-table-providers 465926a3 (spiceai-55, spiceai#80), snowflake-rs f5557381 (spiceai-59, spiceai#13), spark-connect-rs 18ae9bd3 (spiceai-59-2, spiceai#15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (spiceai#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: spiceai#49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. spiceai#68 is still open, waiting on license-compliance decisions. Pin 718429f, the spiceai#73 merge, until spiceai#68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, spiceai#73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of spiceai#68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 718429f, the previous pin (`git diff 718429f a7c4c58` is empty). Repoint the pin at the landed revision and drop the TEMPORARY marker from the fork-patch ledger, as scripts/check_fork_patches.py asks. The ledger's ballista section now names spiceai-55 and upstream 55.0.0-rc1 as its base. Cargo.lock changes only the source revision of the five ballista packages. * perf(cayenne): keep small Vortex scans whole on the main query path DataFusion 55 lowered `repartition_file_min_size` from 10 MiB to 1 MiB (apache/datafusion#22439), so the main Cayenne scan byte-range-splits a 1-10 MiB dimension table `target_partitions` ways and pays a Vortex footer open per range. The main scan now opts small scans out below 10 MiB, as the internal and protected-snapshot scans already do below 256 MiB; Parquet listing scans keep the new default. `small_snapshot_groups_opt_out_of_repartitioning` now asserts the main scan's opt-out and fails with the threshold back at 0. * docs(fork_patches): record duckdb-rs on the protected spiceai-1.4.4-patches-2 head, which table-providers pins * test(duckdb): build CreateExternalTable with DataFusion 55's locations field * ci: validate the datafusion-table-providers pin against spiceai-55, where spiceai#80 landed * test(cayenne): size the point-lookup fan-out control above the main scan's 10 MiB opt-out * fix(data-accelerator-api): implement DataFusion 55's apply_expressions for KeepFirstExec * test: port trunk's new tests to DataFusion 55 (CreateExternalTable locations, common::TableReference) * fix(cayenne): keep -0.0 and 0.0 equal in pushed-down float comparisons and memory-mode indexes DataFusion 55 compares floats with the two zeros equal (apache/datafusion#22835), while Vortex and the Arrow row encoding order them apart. Spell out both zeros when a float comparison or IN list with a zero literal is pushed into a Vortex scan, keep float column-vs-column comparisons above the scan, and hash memory-mode index keys with the file-mode KeyEncoder. * Fix lint --------- Co-authored-by: Luke Kim <[email protected]>

…n fixes backported from 54 (#14800) * build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing #237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for #237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (#70, #71, #72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (#82, #53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, #54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, #15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, #16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, #7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, #50), datafusion-table-providers 465926a3 (spiceai-55, #80), snowflake-rs f5557381 (spiceai-59, #13), spark-connect-rs 18ae9bd3 (spiceai-59-2, #15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: #49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. #68 is still open, waiting on license-compliance decisions. Pin 718429f, the #73 merge, until #68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, #73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of #68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 718429f, the previous pin (`git diff 718429f a7c4c58` is empty). Repoint the pin at the landed revision and drop the TEMPORARY marker from the fork-patch ledger, as scripts/check_fork_patches.py asks. The ledger's ballista section now names spiceai-55 and upstream 55.0.0-rc1 as its base. Cargo.lock changes only the source revision of the five ballista packages. * perf(cayenne): keep small Vortex scans whole on the main query path DataFusion 55 lowered `repartition_file_min_size` from 10 MiB to 1 MiB (apache/datafusion#22439), so the main Cayenne scan byte-range-splits a 1-10 MiB dimension table `target_partitions` ways and pays a Vortex footer open per range. The main scan now opts small scans out below 10 MiB, as the internal and protected-snapshot scans already do below 256 MiB; Parquet listing scans keep the new default. `small_snapshot_groups_opt_out_of_repartitioning` now asserts the main scan's opt-out and fails with the threshold back at 0. * docs(fork_patches): record duckdb-rs on the protected spiceai-1.4.4-patches-2 head, which table-providers pins * test(duckdb): build CreateExternalTable with DataFusion 55's locations field * ci: validate the datafusion-table-providers pin against spiceai-55, where #80 landed * test(cayenne): size the point-lookup fan-out control above the main scan's 10 MiB opt-out * fix(data-accelerator-api): implement DataFusion 55's apply_expressions for KeepFirstExec * test: port trunk's new tests to DataFusion 55 (CreateExternalTable locations, common::TableReference) * fix(cayenne): keep -0.0 and 0.0 equal in pushed-down float comparisons and memory-mode indexes DataFusion 55 compares floats with the two zeros equal (apache/datafusion#22835), while Vortex and the Arrow row encoding order them apart. Spell out both zeros when a float comparison or IN list with a zero literal is pushed into a Vortex scan, keep float column-vs-column comparisons above the scan, and hash memory-mode index keys with the file-mode KeyEncoder. * build: move to iceberg-rust 0.11.0 (RC) and opendal 0.58 Points the iceberg crates at spiceai/iceberg-rust#55, which merges the upstream 0.11.x release branch onto the DataFusion 55 fork line. opendal moves to 0.58 to match iceberg-storage-opendal, so one opendal stack is built instead of two. 0.58 removes the OperatorBuilder step: Operator::new returns the operator with the same default layers that finish() applied. * Bump datafusion to 55.2-rc1 * build: pin iceberg-rust at spiceai-0.11.0-df-55 (6317d03e) spiceai/iceberg-rust#55 merged into spiceai-0.11.0-df-55. Same Rust code as the previous pin (e02547cb); the branch adds only a Python CI pin and a doc-comment fix. fork_patches.md records the new branch and that SigV4 is now an AuthManager in the fork's REST catalog. * build: keep the lockfile's unrelated resolutions from the DataFusion bump The previous commit's lockfile update also re-resolved five crates with range requirements (hashbrown, heck, base64). Restore them so the iceberg repin changes only the iceberg-rust source lines. * build: pin iceberg-rust at spiceai-0.11.0-df-55 (7735caa2) Picks up spiceai/iceberg-rust#49 (IcebergTableProvider::catalog and ::table_ident) and #56 (the fork's own DataFusion pin, which does not affect this build). #49's fork_patches row and its compile guard land with #14588, which first calls the getters. * build: pin datafusion at spiceai/datafusion#250 (55.2.0-rc1 and the 54 backports) Moves the DataFusion fork from 02550cf9 (55.1.0) to 1749ce4e, the head of spiceai/datafusion#250, which merges into #249: `spiceai-55` at 55.2.0-rc1 (spiceai/datafusion#248) plus the `spiceai-54` patches #233, #234 and #245 carried onto 55, and #250's review fix. The guards for those follow in the next commit. The pin names a pull request branch, so the ledger row is marked TEMPORARY and `scripts/check_fork_patches.py` refuses to let it land until #250 and #249 merge and the pin moves to the `spiceai-55` merge commit. The 55.2.0-rc1 merge changes no Spice patch. Outside `Cargo.lock`, the fork's 02550cf9..27351ab0 diff has the same patch-id as upstream's 55.1.0..55.2.0-rc1 diff, except `datafusion-spark`'s `quote.rs`, which the fork's backport had already made identical to 55.2.0-rc1's. That backport's ledger row is dropped; upstream carries the fix as apache/datafusion#25277. `Cargo.lock` changes only the 36 DataFusion entries. * test: guard the DataFusion unparser and footer-cache fixes backported from 54 Each backported fix gets a guard here, so the next re-cut of the fork cannot drop it silently, plus a ledger row naming the guard. - spiceai/datafusion#233 (#14373): a join that is another join's right input stays a parenthesised joined table on that join's right. `a_join_that_is_another_joins_right_input_stays_on_its_right` checks that `b` is introduced before the outer `ON` names it and that a LEFT JOIN keeps a filter from inside it out of `WHERE`. With the `sqlite` feature it runs the SQL on SQLite and compares the rows with DataFusion's for the same plan. - spiceai/datafusion#234 (#14375): a `Limit` that is a join input bounds that input, not the join. `a_limit_on_a_join_input_bounds_that_input_rather_than_the_join`, with the same row comparison. - spiceai/datafusion#250: a projection over a join that passes up a mark join's mark is refused rather than unparsed with the mark unbound. `a_mark_a_projection_over_a_join_passes_up_is_refused`. - spiceai/datafusion#245 (#12952): the file metadata cache frees an evicted footer's hit counter. 55's `DefaultCache` already does, so 55 carries no patch; `crates/cayenne/tests/footer_cache_hit_counter_test.rs` pins the behaviour by counting the live heap across 20,000 cold footer reads into a full cache. At 02550cf9 the three unparser guards fail. SQLite rejects the nested join's SQL ("ON clause references tables to its right"), returns 1 row where the plan returns 2 for `b FULL JOIN (c LIMIT 1)`, and the mark join renders as `… LEFT OUTER JOIN (b) ON a.id = b.id WHERE NOT c.mark`. Against a local DataFusion build with the hit-counter prune removed, the footer-cache guard measures 2,646,176 bytes of heap growth. At 1749ce4e all four pass, and the footer-cache growth is 0 bytes. * build: pin datafusion at spiceai/datafusion#249's merge commit Moves the fork pin from #250's branch head to bebc4d58, the merge of #249 into `spiceai-55`. Its tree is #249's head, cbda233a: 55.2.0-rc1 plus the backported #233, #234 and #245. The pin names a landed revision, so the TEMPORARY marker comes off and `scripts/check_fork_patches.py` passes. #250 is not part of this revision, so its guard and ledger row are removed. * Update datafusion version to 55.2 in Cargo.toml * docs(fork_patches): record the unparser frame refactor #249 carries `cbda233a6` is the fourth Spice commit at the pinned revision. It moves the join-input bookkeeping that #233 and #234 add out of `select_to_sql_recursively_inner`, so an unoptimised build's frame stays where it was. It had no ledger row, so a re-cut could drop it without the audit noticing. Its row is a GAP, with an Open gaps entry. What it buys is stack headroom in a debug build: without it, the fork's `roundtrip_statement` needs 2,112 KiB and overflows a 2 MiB test thread. This repo runs tests on 8 MiB threads, so no test here sees the difference deterministically. The entry names the fork-side check to run at a re-cut instead. * Fix tests * build: pin DataFusion at spiceai/datafusion#251, so MAX of a partition column skips empty partitions (#14801) * build: pin datafusion at spiceai/datafusion#251's merge commit Moves the DataFusion fork pin from #249's merge commit to the head of `spiceai-55`, which adds two patches on top of it: - #251: a file that holds no rows gets no partition-column min/max. Without it, `MIN`/`MAX` of a partition column is answered from the listing's statistics with the value of a partition whose files are all empty: `max(p)` answers '99' for a `p=99` directory holding one empty file, where the rows give '3'. - #240: the listing prunes files by metadata-column predicates before opening them and reports those filters `Exact` (#229 on `spiceai-54`). Neither changes a manifest, so `Cargo.lock` only moves its source revision. Each patch gets a ledger row and a guard in the listing connector's tests, which the gate runs as library tests. * fix(listing): keep every predicate residual on the `_location` fast path The pin now carries spiceai/datafusion#240, so the inner listing reports a metadata-column predicate such as `_size < 50` `Exact`. `MetadataPruningListingTable` handed pushdown classification to the inner listing whenever a `_location` predicate was present, but its `_location` fast path opens the named objects and applies no other predicate, so `_location = x AND _size < 50` returned the rows of a file the `_size` predicate excludes. Report every predicate `Inexact` on that path as well, so a residual `FilterExec` re-applies it. This is #14790's change, which makes the same fix for partition predicates, taken byte for byte so that whichever of the two lands second merges cleanly. The new test goes through `create_listing_table` with `_location` and `_size` enabled. * test(metadata): expect the listing to answer `_size = 2319` exactly With spiceai/datafusion#240 the listing prunes files by a metadata-column predicate and reports it `Exact`, so the plan for `met_size`, which enables only `_size`, has no residual `Filter`. The scan lists only the four 2319-byte files; `data_0.parquet` is 2317 bytes, as its S3 `Content-Length` shows. Regenerated by running `metadata::s3_metadata_columns` and accepted with `cargo insta accept`. The rest of the test passes unchanged at this pin, including the `_location` plans and the returned rows. * fix(data-connector-api): satisfy pedantic clippy in the metadata-pruning listing test `make lint-rust`'s `--tests` pass rejected two lints in `listing_table_prunes_files_by_a_metadata_column_predicate`: `redundant_closure_for_method_calls` (`|group| group.len()`) and `format_collect` (`map(format!).collect::<String>()`). Use `FileGroup::len` and build the file contents with `writeln!` into one `String`. The test is unchanged in what it checks. --------- Co-authored-by: Luke Kim <[email protected]> Co-authored-by: Sergei Grebnov <[email protected]>

…n fixes backported from 54 (spiceai#14800) * build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a8; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing spiceai#237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for spiceai#237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (spiceai#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (spiceai#70, spiceai#71, spiceai#72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (spiceai#82, spiceai#53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, spiceai#54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, spiceai#15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, spiceai#16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, spiceai#7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, spiceai#50), datafusion-table-providers 465926a3 (spiceai-55, spiceai#80), snowflake-rs f5557381 (spiceai-59, spiceai#13), spark-connect-rs 18ae9bd3 (spiceai-59-2, spiceai#15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (spiceai#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: spiceai#49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. spiceai#68 is still open, waiting on license-compliance decisions. Pin 718429f, the spiceai#73 merge, until spiceai#68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, spiceai#73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of spiceai#68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 718429f, the previous pin (`git diff 718429f a7c4c58` is empty). Repoint the pin at the landed revision and drop the TEMPORARY marker from the fork-patch ledger, as scripts/check_fork_patches.py asks. The ledger's ballista section now names spiceai-55 and upstream 55.0.0-rc1 as its base. Cargo.lock changes only the source revision of the five ballista packages. * perf(cayenne): keep small Vortex scans whole on the main query path DataFusion 55 lowered `repartition_file_min_size` from 10 MiB to 1 MiB (apache/datafusion#22439), so the main Cayenne scan byte-range-splits a 1-10 MiB dimension table `target_partitions` ways and pays a Vortex footer open per range. The main scan now opts small scans out below 10 MiB, as the internal and protected-snapshot scans already do below 256 MiB; Parquet listing scans keep the new default. `small_snapshot_groups_opt_out_of_repartitioning` now asserts the main scan's opt-out and fails with the threshold back at 0. * docs(fork_patches): record duckdb-rs on the protected spiceai-1.4.4-patches-2 head, which table-providers pins * test(duckdb): build CreateExternalTable with DataFusion 55's locations field * ci: validate the datafusion-table-providers pin against spiceai-55, where spiceai#80 landed * test(cayenne): size the point-lookup fan-out control above the main scan's 10 MiB opt-out * fix(data-accelerator-api): implement DataFusion 55's apply_expressions for KeepFirstExec * test: port trunk's new tests to DataFusion 55 (CreateExternalTable locations, common::TableReference) * fix(cayenne): keep -0.0 and 0.0 equal in pushed-down float comparisons and memory-mode indexes DataFusion 55 compares floats with the two zeros equal (apache/datafusion#22835), while Vortex and the Arrow row encoding order them apart. Spell out both zeros when a float comparison or IN list with a zero literal is pushed into a Vortex scan, keep float column-vs-column comparisons above the scan, and hash memory-mode index keys with the file-mode KeyEncoder. * build: move to iceberg-rust 0.11.0 (RC) and opendal 0.58 Points the iceberg crates at spiceai/iceberg-rust#55, which merges the upstream 0.11.x release branch onto the DataFusion 55 fork line. opendal moves to 0.58 to match iceberg-storage-opendal, so one opendal stack is built instead of two. 0.58 removes the OperatorBuilder step: Operator::new returns the operator with the same default layers that finish() applied. * Bump datafusion to 55.2-rc1 * build: pin iceberg-rust at spiceai-0.11.0-df-55 (6317d03e) spiceai/iceberg-rust#55 merged into spiceai-0.11.0-df-55. Same Rust code as the previous pin (e02547cb); the branch adds only a Python CI pin and a doc-comment fix. fork_patches.md records the new branch and that SigV4 is now an AuthManager in the fork's REST catalog. * build: keep the lockfile's unrelated resolutions from the DataFusion bump The previous commit's lockfile update also re-resolved five crates with range requirements (hashbrown, heck, base64). Restore them so the iceberg repin changes only the iceberg-rust source lines. * build: pin iceberg-rust at spiceai-0.11.0-df-55 (7735caa2) Picks up spiceai/iceberg-rust#49 (IcebergTableProvider::catalog and ::table_ident) and spiceai#56 (the fork's own DataFusion pin, which does not affect this build). spiceai#49's fork_patches row and its compile guard land with spiceai#14588, which first calls the getters. * build: pin datafusion at spiceai/datafusion#250 (55.2.0-rc1 and the 54 backports) Moves the DataFusion fork from 02550cf9 (55.1.0) to 1749ce4e, the head of spiceai/datafusion#250, which merges into spiceai#249: `spiceai-55` at 55.2.0-rc1 (spiceai/datafusion#248) plus the `spiceai-54` patches spiceai#233, spiceai#234 and spiceai#245 carried onto 55, and spiceai#250's review fix. The guards for those follow in the next commit. The pin names a pull request branch, so the ledger row is marked TEMPORARY and `scripts/check_fork_patches.py` refuses to let it land until spiceai#250 and spiceai#249 merge and the pin moves to the `spiceai-55` merge commit. The 55.2.0-rc1 merge changes no Spice patch. Outside `Cargo.lock`, the fork's 02550cf9..27351ab0 diff has the same patch-id as upstream's 55.1.0..55.2.0-rc1 diff, except `datafusion-spark`'s `quote.rs`, which the fork's backport had already made identical to 55.2.0-rc1's. That backport's ledger row is dropped; upstream carries the fix as apache/datafusion#25277. `Cargo.lock` changes only the 36 DataFusion entries. * test: guard the DataFusion unparser and footer-cache fixes backported from 54 Each backported fix gets a guard here, so the next re-cut of the fork cannot drop it silently, plus a ledger row naming the guard. - spiceai/datafusion#233 (spiceai#14373): a join that is another join's right input stays a parenthesised joined table on that join's right. `a_join_that_is_another_joins_right_input_stays_on_its_right` checks that `b` is introduced before the outer `ON` names it and that a LEFT JOIN keeps a filter from inside it out of `WHERE`. With the `sqlite` feature it runs the SQL on SQLite and compares the rows with DataFusion's for the same plan. - spiceai/datafusion#234 (spiceai#14375): a `Limit` that is a join input bounds that input, not the join. `a_limit_on_a_join_input_bounds_that_input_rather_than_the_join`, with the same row comparison. - spiceai/datafusion#250: a projection over a join that passes up a mark join's mark is refused rather than unparsed with the mark unbound. `a_mark_a_projection_over_a_join_passes_up_is_refused`. - spiceai/datafusion#245 (spiceai#12952): the file metadata cache frees an evicted footer's hit counter. 55's `DefaultCache` already does, so 55 carries no patch; `crates/cayenne/tests/footer_cache_hit_counter_test.rs` pins the behaviour by counting the live heap across 20,000 cold footer reads into a full cache. At 02550cf9 the three unparser guards fail. SQLite rejects the nested join's SQL ("ON clause references tables to its right"), returns 1 row where the plan returns 2 for `b FULL JOIN (c LIMIT 1)`, and the mark join renders as `… LEFT OUTER JOIN (b) ON a.id = b.id WHERE NOT c.mark`. Against a local DataFusion build with the hit-counter prune removed, the footer-cache guard measures 2,646,176 bytes of heap growth. At 1749ce4e all four pass, and the footer-cache growth is 0 bytes. * build: pin datafusion at spiceai/datafusion#249's merge commit Moves the fork pin from spiceai#250's branch head to bebc4d58, the merge of spiceai#249 into `spiceai-55`. Its tree is spiceai#249's head, cbda233a: 55.2.0-rc1 plus the backported spiceai#233, spiceai#234 and spiceai#245. The pin names a landed revision, so the TEMPORARY marker comes off and `scripts/check_fork_patches.py` passes. spiceai#250 is not part of this revision, so its guard and ledger row are removed. * Update datafusion version to 55.2 in Cargo.toml * docs(fork_patches): record the unparser frame refactor spiceai#249 carries `cbda233a6` is the fourth Spice commit at the pinned revision. It moves the join-input bookkeeping that spiceai#233 and spiceai#234 add out of `select_to_sql_recursively_inner`, so an unoptimised build's frame stays where it was. It had no ledger row, so a re-cut could drop it without the audit noticing. Its row is a GAP, with an Open gaps entry. What it buys is stack headroom in a debug build: without it, the fork's `roundtrip_statement` needs 2,112 KiB and overflows a 2 MiB test thread. This repo runs tests on 8 MiB threads, so no test here sees the difference deterministically. The entry names the fork-side check to run at a re-cut instead. * Fix tests * build: pin DataFusion at spiceai/datafusion#251, so MAX of a partition column skips empty partitions (spiceai#14801) * build: pin datafusion at spiceai/datafusion#251's merge commit Moves the DataFusion fork pin from spiceai#249's merge commit to the head of `spiceai-55`, which adds two patches on top of it: - spiceai#251: a file that holds no rows gets no partition-column min/max. Without it, `MIN`/`MAX` of a partition column is answered from the listing's statistics with the value of a partition whose files are all empty: `max(p)` answers '99' for a `p=99` directory holding one empty file, where the rows give '3'. - spiceai#240: the listing prunes files by metadata-column predicates before opening them and reports those filters `Exact` (spiceai#229 on `spiceai-54`). Neither changes a manifest, so `Cargo.lock` only moves its source revision. Each patch gets a ledger row and a guard in the listing connector's tests, which the gate runs as library tests. * fix(listing): keep every predicate residual on the `_location` fast path The pin now carries spiceai/datafusion#240, so the inner listing reports a metadata-column predicate such as `_size < 50` `Exact`. `MetadataPruningListingTable` handed pushdown classification to the inner listing whenever a `_location` predicate was present, but its `_location` fast path opens the named objects and applies no other predicate, so `_location = x AND _size < 50` returned the rows of a file the `_size` predicate excludes. Report every predicate `Inexact` on that path as well, so a residual `FilterExec` re-applies it. This is spiceai#14790's change, which makes the same fix for partition predicates, taken byte for byte so that whichever of the two lands second merges cleanly. The new test goes through `create_listing_table` with `_location` and `_size` enabled. * test(metadata): expect the listing to answer `_size = 2319` exactly With spiceai/datafusion#240 the listing prunes files by a metadata-column predicate and reports it `Exact`, so the plan for `met_size`, which enables only `_size`, has no residual `Filter`. The scan lists only the four 2319-byte files; `data_0.parquet` is 2317 bytes, as its S3 `Content-Length` shows. Regenerated by running `metadata::s3_metadata_columns` and accepted with `cargo insta accept`. The rest of the test passes unchanged at this pin, including the `_location` plans and the returned rows. * fix(data-connector-api): satisfy pedantic clippy in the metadata-pruning listing test `make lint-rust`'s `--tests` pass rejected two lints in `listing_table_prunes_files_by_a_metadata_column_predicate`: `redundant_closure_for_method_calls` (`|group| group.len()`) and `format_collect` (`map(format!).collect::<String>()`). Use `FileGroup::len` and build the file contents with `writeln!` into one `String`. The test is unchanged in what it checks. --------- Co-authored-by: Luke Kim <[email protected]> Co-authored-by: Sergei Grebnov <[email protected]>

…iceai#14844) * feat(cluster): add runtime.executor.task_slots to override executor task slots * feat(cluster): honor runtime.executor on the executor, decouple task-runner threads from task slots * test(cluster): cover runtime.executor.task_slots end to end Start a real scheduler and executor and read the slot count the executor registered with the scheduler, for the scheduler-supplied value, an executor-own override, and the CPU budget default. Link the task_slots validation errors to the runtime reference page and record the datafusion-ballista pin's branch accurately. * test(cluster): guard task-runner pool size against the slot count * ci: run the task-runner thread guard in the integration workflow * revert: drop runtime.executor.task_slots and the executor changes (out of scope for #14845 milestone 1) * feat(cayenne): keep one row per primary key automatically and deprecate on_conflict (#14726) * fix(cayenne): reject duplicate primary keys in full refresh Fixes #14578 * fix(cayenne): honor overwrite dedup modes across batches * fix(cayenne): honor drop conflicts during full refresh * refactor(cayenne): resolve repeated keys with a per-table conflict policy Replace the whole-stream overwrite validation and the shared dedup crate's buffered overwrite mode with a Cayenne-owned policy that follows the documented on_conflict semantics: each incoming record batch is one upsert statement, so a key repeated within a batch follows the dataset's upsert refinement and a key repeated across batches is a later upsert. This commit adds the policy (key_conflicts) and threads the upsert refinement (upsert_dedup / upsert_dedup_by_row_id) from the accelerator into the table and its partitions. The write paths adopt it in the following commits. * fix(cayenne): keep the last copy of a key repeated across a full refresh's batches A file-backed full refresh stored every copy of a primary key its source repeated across record batches (#14578). Under the upsert policies a later batch's copy is a later upsert of that key, so the refresh now resolves it the way separate upserts would: a layer splitter routes each batch into the main snapshot or into protected layers above it, starting a new layer when a batch repeats a key the current layer may hold (or the layer reaches 8M rows), and tombstoning, in each layer above the first, the keys an earlier layer may hold. Both checks use bloom filters charged to the query memory pool; a false positive costs a layer or a tombstone, never a row. The layers, their key tombstones and their thresholds commit in the overwrite's own transaction and publish in its visibility flip, leaving exactly the state an overwrite followed by one upsert per layer leaves. Under drop the first copy is kept with an exact key set. * Resolve repeated keys before buffered Cayenne writes * test(runtime): cover Cayenne on_conflict for keys repeated within and across batches Runs every on_conflict policy through the runtime in mode: memory and mode: file, for full and append refreshes and for a partitioned table, with a key repeated past the first record batch and within one batch. Replaces the narrower Cayenne cases added to the DuckDB-gated on_conflict_options tests. * docs(cayenne): describe how a write resolves keys it repeats * Resolve repeated keys in streaming Cayenne appends atomically * fix(cayenne): size layer key filters to the write and charge appends to their query The layer splitter's first bloom filter held 64K keys (384 KiB at 48 bits) however small the write, and a streaming append charged it to the table's runtime pool rather than the pool of the query running the write. Start at 1K keys and grow each next filter fourfold (eight filters cover a full layer), and register an append's reservation on its task context. * perf(cayenne): keep one tombstone per key for a layered refresh and build its index once A refresh that repeats every key across N layers recorded N-1 tombstones per key and folded each layer into the live deletion index under the listing fence. Tombstones merge by maximum and a layer tombstones only keys it holds, so only a key's highest tombstone matters: keep that one, and build the new snapshot's index outright from the layers before the fence. The per-layer key check uses an exact set of 64-bit key hashes; a chain of bloom filters summed their false-positive rates and split refreshes that repeat no key. Adds a release-mode benchmark (examples/layered_refresh_bench.rs). * perf(cayenne): collapse a bounded window of a streaming upsert write before layering it A streaming full refresh or append now resolves the keys a 128 MiB window of its input repeats in memory (last copy wins) before the layer splitter sees it, so a repeat inside the window opens no layer and leaves no tombstone. Batches are resolved as they arrive, keeping the work per poll to one batch, and the window is charged to the query memory pool. * perf(cayenne): hide a position-deletion table's superseded refresh copies by position On a table that deletes by position, a full refresh whose input repeats keys across its collapse windows now writes every layer into the main snapshot and hides the copies a later layer supersedes with position deletes, found by a read-back of the lower layers' key columns. The refresh publishes one snapshot with no protected layer and no key tombstone, reads skip the deleted pages, and the live row count stays exact. Key-deletion tables keep the protected layers and key tombstones. The read-back lists files without statistics: read before the deletes are installed, they would be cached counting the hidden rows. Position capture casts the key columns Vortex returns in view form to the table's key type, and skips rows a position delete already hides. * docs(cayenne): note that repeated keys are resolved within each partition * perf(cayenne): fold a key-deletion refresh's superseded copies out of its files before publishing A refresh that repeats a key across collapse windows wrote the later copies as protected layers with key tombstones, which made scans and point lookups read every layer until compaction (p99 full scan 51.9 ms, lookup 33.9 ms at 1M keys x 8 copies). Write every layer into the one snapshot instead, locate the superseded copies by a read-back of their keys, and on a table that deletes by key rewrite the files holding them without them. The refresh then publishes one snapshot with no deletes and exact statistics, rebuilt from per-file statistics recorded during the write. A table that deletes by position keeps hiding the copies with position deletes, but its statistics are now served inexact while those deletes are pending: they still describe the hidden rows, and after a reopen MAX() answered from them returned a superseded copy's value. * fix(cayenne): keep the last copy of a key a transaction's write repeats across batches A write inside BEGIN ... COMMIT failed with 'Incoming data contains duplicate primary key across batches' whenever its input repeated a key across record batches, although under the documented on_conflict semantics each batch is one upsert and the later copy wins. Resolve repeats within a batch before validation, validate a repeat across batches as a later copy that supersedes the stored row once, write the layers in order into the staged snapshot, and fold the superseded copies out before the commit. * style(cayenne): format the staged-upsert repeat test * refactor(cayenne): share the layered fold between refreshes and transactions Write the later layers and fold them through one step used by both a full refresh and a transaction's staged snapshot, and derive the live row counts from the statistics rather than threading them through. Pass the cross-batch repeat handling explicitly from the writer that folds, delete superseded files concurrently while releasing their cached footers, log a failed cleanup of an unpublished snapshot directory, and share the NDV sketch merge. * perf(cayenne): locate superseded refresh copies as they are written Route every layered write through one exact map from each admitted key's 128-bit digest to its layer and, once the write observer sees it land, its file and position. A later layer admitting the key again records the earlier copy's position directly, so a refresh no longer reads its lower layers' keys back to find the copies they supersede, needs no bloom filter or per-layer key set, and no longer cuts layers at a row cap. The map is pre-sized from the replaced table's row count and charged to the query memory pool; a row the observer cannot locate fails the refresh. * perf(cayenne): skip the collapse window's filter when it holds no repeated key A window over distinct keys looked every row up again on drain only to keep all of them; it now passes its batches through when no key repeated. At 40M distinct keys this halves the window's cost (0.78 s to 0.41 s). * fix(cayenne): resolve repeated keys only for refresh writes A dataset's on_conflict now resolves the keys a refresh's incoming data repeats; a user's statement keeps its own semantics. The refresh task marks the session it writes through (util::session_state::AcceleratorRefreshWrite), and Cayenne's insert_into resolves repeats only for a write that carries it, so a user INSERT or a write inside BEGIN ... COMMIT behaves as before: a key it repeats across record batches fails it. Change streams, which do not go through insert_into, keep change-stream semantics. * fix(cayenne): mark user statements instead of refresh writes A refresh path that did not mark its session (a localpod parent's MultiSink, say) fell back to storing every copy of a repeated key. Mark the user's statements instead — AcceleratedTable::insert_into and the write-back sink, via util::session_state::UserStatementWrite — and resolve repeats for every other write, so a missed path deduplicates rather than duplicates. Partitioned inserts carry the marker to each partition's table through PartitionContext. * fix(cayenne): bound the collapse window by everything it holds The window filled at COLLAPSE_WINDOW_BYTES of rows but also charged the map of their keys, uncounted, and kept that map's capacity across drains: an 8 MiB window peaked at 16.2 MB and held its map between windows. Count the map toward the bound and release it on every drain. * fix(cayenne): spill a refresh's key map instead of failing when the pool refuses it A refresh with on_conflict reserved query-pool memory for every key it admitted and failed the whole refresh when the pool refused, even with no key repeated (1M keys in a 64 MiB pool, 16M in 1 GiB). Now a refused reservation cuts the layer at the next batch; between layers, with every admitted key located, the map spills to the runtime's spill directory as a run sorted by digest, and after the last layer the runs merge with the map to find the superseded copies (highest layer wins; lowest under drop). drop takes the same path in place of its unbounded exact set, and the collapse window drains early rather than fail when the pool refuses it. * revert(accelerator-cayenne): keep the auto-tuned config log as on trunk An earlier commit on this branch added metastore_dir to the auto-tuned config log line and its re-emit fingerprint, which is unrelated to resolving repeated keys; drop it. Also move upsert_dedup_for out from under scan_view_reuse_for's doc comment. * fix(cayenne): resolve the repeats a partitioned append refresh holds within a window A partition's append is one staged snapshot of the cross-partition append, so it cannot take the layers a streaming append splits a repeated key into, and a key its data repeated across record batches failed the refresh. Put the bounded collapse window in front of it: repeats within a window resolve per on_conflict, and a key repeated further apart is still rejected by validation, as before. Document the remaining limitation and cover the partitioned append refresh in the runtime matrix. * docs(cayenne): describe the window a partitioned append refresh resolves repeats in * fix(cayenne): keep the last copy of a key a refresh repeats within one batch under upsert Conflict validation already keeps the last copy of a key repeated within a batch under a plain `upsert` (`UpsertOptions::last_write_wins`), so an append refresh of such data loaded on trunk. Resolving the refresh's repeats ahead of validation rejected them instead, failing loads that used to succeed. A plain `upsert` now resolves as `upsert_dedup_by_row_id` does: the last copy wins within and across batches, so a refresh's outcome no longer depends on where record-batch boundaries fall. `upsert_dedup` still fails on differing copies within a batch. * chore(cayenne): drop stale fingerprint tests and fix lint on the layered refresh path Remove the auto-tuned config fingerprint tests that still passed a metastore path after that change was reverted, take the map entry once when keeping a key's first copy, make CollapseWindow::push infallible, and pass the partitioned append's user-statement marker through a builder method. * fix(runtime-table): mark a user UPDATE as a user statement UPDATE writes its new rows through the accelerator's own insert, which never passed the marked AcceleratedTable::insert_into, so Cayenne resolved the keys it repeated as it does for a refresh: an UPDATE moving rows from several batches onto one key succeeded and kept one row instead of failing as on trunk. Mark the session once, before choosing a write path, in both insert_into and update. * feat(acceleration): add on_conflict upsert_dedup_by_time_column Keep the row with the greatest time_column per primary key across an acceleration refresh, so out-of-order source data (late files, Iceberg snapshots with older rows) no longer overwrites a newer version. A streaming step in the refresh keeps key -> greatest time and writes a row only when it is strictly newer than every version already kept, collapsing a key's versions within a batch. On append, it is seeded from the acceleration's keys and times inside the refresh_append_overlap window, replacing the unchanged-row check for this mode. Superseded rows are counted in dataset_acceleration_refresh_rows_superseded{reason=older|equal_time}; on append the count includes rows re-read from the overlap window. A NULL time_column fails the refresh with a count, never row values. Available on Cayenne and SQLite, which apply a key repeated across one write's batches in order. DuckDB applies a write as one INSERT ... ON CONFLICT and keeps an unspecified copy of a repeated key, so it is refused at load, along with a missing time_column or declared primary key, a time_column inside the key, and refresh modes other than full and append. Warn when an append has no refresh_append_overlap. * test(runtime): satisfy pedantic lints in the on_conflict batch tests * test(acceleration): integration tests for upsert_dedup_by_time_column Cayenne and SQLite, in memory and file mode: a full refresh over two files whose newest versions of two keys arrive in opposite orders, a key repeated within a batch, and a filler spanning several batches; an append where a late, older row inside the overlap must not replace the stored version and a newer row must; and a NULL time_column failing the refresh. With plain on_conflict: upsert all three fail. * refactor(cayenne): share the collapse pump and move spill work off the async thread - One Collapser resolves input batches and a window's repeats for both the layer source and the partitioned append's CollapseStream. - The survivor comes from the resolver's policy (Survivor::for_policy); the window takes it instead of a separate keep-first flag. - A drop overwrite takes no window: the splitter already drops a later copy as it admits it. - An overwrite's spill directory is part of its split mode, so there is no unspillable overwrite to fail. - An append clears its map when a routed batch starts a layer, so the admitted-layer field goes. - Spilling and taking the superseded copies swap the map under the lock and sort, write, merge and bucket on the blocking pool; merge sources are iterators, and spill I/O uses 1 MiB buffers. - for_user_statement takes the marker, so each partition sink is one call. - Docs: drop's path, the spill, and the window's users. * proto(cayenne): resolve a full refresh's repeated keys after the write Prototype, behind SPICE_CAYENNE_REFRESH_DEDUP=postpass: a full refresh writes once into its new snapshot, each batch resolving its own repeats (a vectorized hash check first, the exact resolve only for a batch that may repeat a key) and stamping its rows with the batch's arrival sequence in a trailing u32 column the table schema does not name. After the write, a query over the files' key columns, ordinals and row positions, run in hash chunks of about 1 GiB of written data, finds every copy other than the one the policy keeps; position mode hides them with position deletes, key mode folds them out of the files that hold them. The layered refresh bench gains --key-kind (int, string, composite), --dup-fraction, --progress-rows, --memory-limit-mb, --skip-queries and --sql. * proto(cayenne): resolve a streaming append's repeated keys after the write Prototype, behind SPICE_CAYENNE_REFRESH_DEDUP=postpass: an unpartitioned streaming append writes once into a new protected snapshot, each batch resolving its own repeats and stamping its rows with its arrival sequence. Conflict validation still supersedes (upsert) or keeps (drop) the stored copy an incoming key meets, but keeps a key the append repeats across batches; only the key's first copy records the stored copy, so the live row count counts it once. After the write, the duplicate query over the new snapshot's files alone finds the incoming copies the policy does not keep: position-deletion tables hide them with position deletes published with the snapshot, key-deletion tables fold them out of the files. A write of one batch skips the query. The layered refresh bench gains --append. * proto(cayenne): size the post-pass duplicate query's chunks by rows written The query's memory follows the rows a chunk covers, not the bytes they were written in. A well-compressed table packs many rows into each GiB written, so chunks sized from bytes alone held tens of millions of rows each. Chunks now cover at most 16 Mi rows (override: SPICE_CAYENNE_REFRESH_DEDUP_CHUNK_ROWS), and still at most one CHUNK_BYTES of written data. * proto(cayenne): chunk a refresh's duplicate query by written key ranges Behind SPICE_CAYENNE_REFRESH_DEDUP_CHUNKING=range, the post-pass reads each written file's footer bounds on the key columns, groups files whose bounds overlap, and gives each step of the duplicate query only its own files (hash-slicing a group larger than a chunk), instead of reading every file for every hash chunk. Overlapping files fall back to the hash chunks. SPICE_CAYENNE_REFRESH_DEDUP_SPLIT=1 splits a step's files into row ranges when it has fewer files than partitions. SPICE_CAYENNE_POSTPASS_DEBUG=1 prints per-file bounds, the plan, per-step timings and the pool peak. The bench gains --key-order sorted. * proto(cayenne): split a large key-range cluster into key sub-ranges pushed into the scan Behind SPICE_CAYENNE_REFRESH_DEDUP_SUBSPLIT=range: a cluster of files larger than a chunk is cut into equal-width sub-ranges of an integer key column, each step reading the cluster's files with a key-range filter the scan prunes zones by, rather than hash slices that each read the whole cluster. Also passes the written row count to the streaming append's duplicate query after the merge. * proto(cayenne): cut key ranges at sampled quantiles for non-integer leading keys Key sub-ranges become half-open [lo, hi) bounds of any key type that tile the whole domain. A cluster on an integer column is cut equal-width from its bounds; one on another column (strings) at quantiles of about 1,024 evenly spaced rows sampled per file. Each split cluster checks that its ranges read every one of its footer-counted rows, and fails the refresh otherwise. * refactor(cayenne): resolve a refresh's repeated keys after the write, by default Make the post-write resolution the only path a refresh that resolves repeated keys takes: a full refresh, and a streaming append on a table without a partition column. The duplicate query always chunks by the written files' key ranges, cutting a large cluster into key ranges pushed into the scan, sized from the rows and bytes written. Remove what it replaces: the overwrite's key map, its write observer, the spill and merge, the layered overwrite and append, and FirstCopyFilter. The collapse window, still used by a partition's staged append, moves to collapse_window.rs, and Survivor to key_conflicts.rs. Remove the prototype switches and diagnostics. The repeated-key tests move to repeated_keys.rs and gain a refresh cut into many key ranges for integer and string keys. * wip(acceleration): upsert_dedup_by_time_column content-hash ties, query-pool spill, Arrow support Settle equal times by a 64-bit content hash, charge the key map to the query memory pool and spill when it is refused, show the mode's refresh errors unwrapped, keep the last copy of a repeated key on Arrow for this mode, and refuse an on_conflict that is not the primary key or a partitioned Arrow acceleration. Includes threading the query RuntimeEnv into refreshes and a warning for upsert with refresh_append_overlap. * test(cayenne): a staged append refuses a keyed table Dual-write reaches Cayenne only through begin_staged_append, which refuses a table with a primary key in either deletion mode, so a user statement through dual-write never reaches the repeated-key resolution a refresh runs. * fix(cayenne): store a refresh's arrival column under a name the table does not use A table with its own column named __cayenne_arrival failed every refresh that resolves repeated keys ("StructLayout must have unique field names"). The arrival column is now stored under the first of __cayenne_arrival, __cayenne_arrival_1, ... the table does not already use, and the read-back reads it by that name. * wip(acceleration): resolve upsert_dedup_by_time_column full refreshes in Cayenne's post-pass A full refresh of an unpartitioned file-mode Cayenne table hands each row's version (time, content hash) to the write instead of resolving repeated keys in a map: Cayenne stamps them in hidden columns as it writes, keeps each key's greatest version within a batch, and its post-pass keeps each key's greatest (time, hash, arrival) across batches. Other engines and append keep the selector. Every refresh write reports the copies of repeated keys the accelerator does not keep, so rows_superseded covers every on_conflict mode on Cayenne, and rows_written counts every row a refresh receives on every path. * wip(cayenne): order a buffered overwrite's repeated keys by row version A full refresh small enough for the inline path now keeps each key's greatest (time, hash) rather than skipping that path: the buffered rows are sorted by version before the existing collapse keeps each key's last copy, and the copies not kept are reported. * wip(acceleration): resolve upsert_dedup_by_time_column versions before a synchronized child sees them - A dataset with a synchronized (localpod) child resolves row versions in the runtime before broadcasting, since a child cannot read them; MultiSink refuses row versions if a child attached mid-refresh. - MultiSink propagates a table's write error instead of discarding it. - An append seed with a string time column within 14 hours of 1970 reads every stored row, since the widened window has no earlier start. - Content hashes normalize -0.0/NaN only for SQLite, which stores them that way; elsewhere an equal hash means identical rows, so the kept copy is independent of arrival order. - Group the refresh's dedup settings into UpsertDedupByTimeColumn; simplify the superseded-metric reporting and the post-pass handoff. * test(cayenne): pass the overwrite's delete files in the turso statement-conflict test Trunk's statement-conflict test calls commit_overwrite with the three arguments it took before this branch added the delete files a refresh commits with its snapshot. * perf(cayenne): load a refresh's first append into an empty table as one unchecked write A refresh appending into a keyed table that holds no rows has no stored key to check its own against. Under the table's write lock, the sink observes every place a row can sit (data files, cold tier, inline rows, the in-memory tier, staged appends) and, when all are empty, writes the load as one append: no segments, no conflict check, and therefore no in-memory set of every incoming key. The keys the load repeats are still resolved after the write, and it publishes as any append does (one protected snapshot), so the table's shape is unchanged. The load records no keys, so it drops any cached primary-key index it did not fill and warms a new one in the background for the next refresh. A user's statement never takes this path. Cayenne benchmark, 100M distinct keys into an empty table: 65.3 s at 6.8 GiB peak RSS (segmented append) becomes 11.2 s at 3.8 GiB. Tests that build their state with plain inserts now mark them as user statements, which they are, so they keep the append path they assert on. * fix(cayenne): a refresh keeps the last copy of a key under every upsert policy Under `upsert_dedup`, a refresh failed when one record batch held two different copies of a key, while the same copies in two batches loaded: whether a refresh succeeded depended on batch boundaries the user cannot see. A refresh now keeps the last copy under `upsert`, `upsert_dedup` and `upsert_dedup_by_row_id` alike, within and across batches, on every path (full, append, partitioned, memory mode, change streams). The accelerator's `UpsertDedupTableProvider` checked each batch before Cayenne saw it, so Cayenne's resolution never ran for those copies. Cayenne now wraps with `wrap_with_statement_upsert_dedup_if_needed`, which deduplicates only a user's statement and passes the accelerator's own writes through. A user's statement keeps statement semantics. Rollback tests that used the old failure as their trigger now use a null primary key, which fails every policy. * perf(cayenne): share the duplicate query's memory budget between concurrent refreshes Each write that resolves its repeated keys after the write sized its duplicate query's chunks for a full budget of its own (16M rows, about 1 GiB). The partitions of a partitioned table resolve their repeats at the same time, so a partitioned full refresh held one full chunk per partition at once. The chunk budget is now shared: a write claims a share when it starts and frees it once its repeats are resolved, and each sizes its chunks for its share. A single refresh keeps the whole budget; eight partitions take an eighth each. Measured with the chunk size swept by hand on an 8-partition full refresh of 100M keys plus 20% repeats: 16M-row chunks peaked at 9.9 and 11.0 GiB, 1M-row chunks at 4.1 and 4.3 GiB (trunk: 3.8-4.4 GiB), with no slower load; an unpartitioned refresh keeps its full-size chunks, which smaller ones would slow by about 12%. * wip(cayenne): order a refresh's first load into an empty table by row version A refresh that supplies row versions (`upsert_dedup_by_time_column`) and appends into an empty table now takes Cayenne's one-write first load with the version columns stamped, and resolves the keys it repeats by version instead of arrival. Any other append that carries row versions is refused: it would supersede a stored copy whatever its version. The runtime hands versions to an append only into an acceleration it observed empty with no retention filter; otherwise its selector resolves them before the write. The first-load integration test now also runs `refresh_mode: append` on every engine, with enough rows that the load streams in many batches. * Integrate snapshot metadata listing with the incremental secondary index * Account for retained key digests and queued collapse batches * Add Cayenne per-statement conflict-policy and failed-append regressions * Apply strict Cayenne upsert semantics across incoming batches * Guard failed Cayenne snapshot publication against durable partial updates * Commit Cayenne append deletions and snapshot visibility together * Align Cayenne lifecycle regressions with explicit conflict policies * Resolve staged Cayenne statements across the whole input * Compare Map and nested dictionary payloads in Cayenne content identity Strict upsert collapses identical copies by a digest of each row's Arrow row encoding. The row format cannot encode Map or dictionaries of nested values, so any table with such a column failed every keyed write. Rewrite those columns as the value IS NOT DISTINCT FROM compares (a map as its entry list, a dictionary as its values) before encoding. A seeded differential test checks the digest against DataFusion's IS NOT DISTINCT FROM on every pair of rows, per column and per row, including signed zero, NaN payloads, NULLs over masked children and repeated dictionary values. The helper-column layout test and the name-collision test now cover both content columns. * Resolve a Cayenne append's failed COMMIT by reading back its outcome A COMMIT that reports an error may still have committed. The resolved append then deleted its snapshot directory and deletion vectors while the catalog referenced them, and the table failed to reopen with a missing deletion vector. Read back the snapshot's sequence row, which commits with the rest of the publication: present, publish; absent, discard; unreadable, keep the files and refuse further writes until the table is reloaded. Tests inject each outcome into the commit and pause a commit to cancel its caller, which must not release the write lock before publication. * Count every Cayenne key with different versions in the whole statement Strict upsert stopped at the first record batch that held different versions of a key, so the count in its error depended on batch boundaries (2, 3 or 4 for the same rows). A buffered write now checks all of its rows before failing, and a streamed write splits a batch's versions of a key into separate batches so the post-write resolution counts each key once across the statement. * Let Cayenne resolve every write's repeated keys itself Cayenne applies on_conflict to refreshes, change streams and user statements alike, so the factory no longer wraps it in the generic statement deduplication provider, and the session marker that told the two kinds of write apart is gone. * Resolve a Cayenne partition's append over its whole input A partition's staged append resolved repeated keys only inside a bounded in-memory window and rejected a key repeated further apart, so whether a refresh succeeded depended on how far apart its batches repeated a key. It now stamps arrival and folds the superseded copies out of its staged files with the same post-write resolution a streaming append uses, and the window is removed. * Resolve Cayenne transaction statements whole and read back a failed commit A multi-table transaction now publishes each table's statement resolved over its own input, skips a table it wrote no rows to instead of publishing an empty snapshot, and reads back each staged snapshot's sequence after a COMMIT reports failure, keeping the staged files when the outcome cannot be determined. Tests drive the real transaction orchestrator across two tables in both deletion modes and inject each commit outcome. * Add upsert_by_arrival and count the rows Cayenne refreshes supersede upsert_by_arrival keeps the version of a key that arrived last; it loads only on Cayenne and other engines refuse it by name. On Cayenne the deprecated aliases upsert_dedup and upsert_dedup_by_row_id warn once at load and name the option they behave as. The strict upsert error no longer mentions an option this release does not have. dataset_acceleration_refresh_rows_superseded{reason} counts the rows a refresh received but did not keep: unchanged for an identical copy, arrival for a version settled by arrival order, including a row drop discards for a stored key. Cayenne counts them into a SupersededRows the refresh attaches to its session; the reasons an option can produce are published at zero at load. A localpod parent's refresh now reports its write's failure instead of succeeding with no change. * Align Cayenne integration tests with the conflict policies Plain upsert now fails on different versions of a key in one statement, so the keep-last tests use upsert_by_arrival and a new test covers the strict failure. A first insert into an empty table builds its key index in the background, which the accounting tests now wait for. * Satisfy rustfmt and the pedantic test lints * Refuse deletes and table maintenance once a write's commit outcome is unknown A table that could not learn whether a write's catalog commit happened refused further appends and staged writes, but statement and retention deletes, MERGE key-probe deletes, inline and mem-tier checkpoints, compaction and cold-tier promotion still ran from its in-memory view, which may miss rows the catalog holds. Each now checks the same guard after taking the write lock, so only reloading the table from its catalog lets it change again. * refactor(cayenne): align conflict policy names with on_conflict semantics * test(cayenne): scope CDC replacement assertions to upserts * Retry the duplicate query with fewer partitions once it runs out of memory The post-write duplicate query retried a ResourcesExhausted failure only by cutting the key space finer. Each partition's hash join and aggregate still reserve a fixed minimum (about 320 KB per join input), so with many partitions and a small pool no chunk size fits, and a retry that started while the failed attempt's tasks still held their memory found the pool full. On a 24-core host the 8 MB-pool test failed 3 of 4 tries in signoff. Each retry now also halves the query's partitions, down to one, and first waits (up to 10 s) until the failed attempt has released the memory it held, counted by a pool wrapper scoped to the query. Under SPICE_CPU_CORES=64 with every core loaded, the bounded-pool test failed 1 of 5 runs before and 0 of 10 after; waiting alone left 5 of 10 failing. * Adapt time-ordered refresh spill files to DataFusion 55.1 DataFusion 55.1 returns spill files as Arc<dyn SpillFile>: write through open_writer(), which accounts disk usage as it writes, and read back by the file's local path. The Arrow IPC writer over a SpillWriter is not Sync, so the deferred-rows writer sits behind a mutex reached only via get_mut(). TableReference moved from datafusion::sql to datafusion::common. * Keep the last arrival for every Cayenne table with a primary key #14576: a primary key alone decides what Cayenne does with a key it sees again, so on_conflict no longer chooses between versions. - A refresh or statement keeps the copy of a repeated key that arrived last, within and across batches. The failing `upsert`, `drop`'s first-copy rule, and the row-content digest that told identical copies from versions are gone, with the UpsertPolicy plumbing that selected between them. - A table opened with any on_conflict upserts on its primary key, so a table an earlier release created with `drop` (DoNothing) keeps the last version too. pk_conflict_detection: none still leaves a table without conflict resolution. - Cayenne's table creation ignores the on_conflict values and their validation. The setting stays on the acceleration because it still keeps a read-write dataset's writes in the acceleration. - Every copy a write supersedes counts as `arrival`, seeded at zero for a dataset with a primary key. * Order a key's versions by time_column whenever one is set #14576: a Cayenne acceleration with a primary key and a time_column keeps the newest version of each key by that column, without an on_conflict value asking for it. - A full or append refresh orders versions by time_column whenever the dataset sets one. It keeps the last arrival instead when the table has no primary key, its key holds the time column, or the rows the refresh reads lack it, so no configuration that loads today is refused or fails. - Equal times go to the later arrival, so a correction that keeps its time replaces the row it corrects. A row identical to the kept one is a re-read: it is not written again, and counts as superseded by arrival. - A NULL time is older than any time rather than failing the refresh. - Versions carry only their time: the content hash Cayenne ordered ties by, and its trailing column, are gone. Superseded rows count as `older` or `arrival`. - The load-time refusals of `upsert_by_time` are gone with the field that enabled it; the refresh's internal names follow the rule it implements. * Remove the unreleased upsert_by_arrival and upsert_by_time values #14576 replaces explicit on_conflict values with a rule inferred from primary_key and time_column, so the two values this branch added never ship: on_conflict is back to drop, upsert, upsert_dedup and upsert_dedup_by_row_id, with the error that refused the new values off Cayenne. The warnings that named them go too: the one calling upsert_dedup and upsert_dedup_by_row_id aliases of the new values, and the one recommending upsert_by_time for an append that re-reads an overlap. The no-overlap warning now describes the inferred rule instead of the value. * State each Cayenne dataset's key rule at load and deprecate on_conflict #14576's load lines: - A Cayenne dataset with a primary key logs the rule it keeps one row per key by: the newest by time_column (the later arrival on a tie), the last arrival (noting that it can differ between refreshes, and that a time_column makes it reproducible), or each change in order for refresh_mode: changes. - A dataset that sets on_conflict is warned that it is deprecated. On Cayenne the warning says it is no longer used and names any change in which version a key keeps; on another accelerator it says on_conflict is removed in 3.0 and that Cayenne keeps one row per key without it. A read-write dataset is not warned: on_conflict still keeps its writes in the acceleration. - The Spicepod schema marks on_conflict deprecated. * Report rows a refresh superseded as dataset_acceleration_rows_superseded A Cayenne table keeps one row per primary key, so the rows a refresh receives but does not keep are counted under one `dataset_acceleration_rows_superseded` counter with the two reasons #14576 specifies: `older` (a later time_column was kept) and `arrival` (a later arrival was kept). Counting a user's statements follows separately. * Describe Cayenne's primary key rules in the technical reference The repeated-keys section still described the on_conflict policies, the content digest, and the refresh_rows_superseded metric. State the #14576 rules instead: last arrival, newest by time_column, change order; and dataset_acceleration_rows_superseded with its older and arrival reasons. * Let a memory-mode Cayenne table order a full refresh's versions itself A memory-mode full refresh of a dataset with a primary key and time_column ran the refresh's own newest-by-time selection over every row, then Cayenne resolved the same keys again over the buffered write. Cayenne's buffered write already orders a key's copies by the row versions a refresh supplies, so a memory-mode overwrite now gets the versions, as a file-mode one does, and the refresh no longer selects. An append into an empty table still selects in memory mode, which only file mode resolves after writing. Release spiced, 5M rows over 2.5M keys, memory mode, full refresh, 8 refreshes per run, arms interleaved, same rig: P99 681 ms -> 389 ms (P50 592 -> 353 ms); the selection was 35.8% of busy CPU in a profile of the old path. File mode runs the same code in both arms (P99 840 vs 857 ms). Every run kept exactly one row per key, the newest. * Count the rows a statement supersedes, and pin its row count #14576 counts a user's statement under dataset_acceleration_rows_superseded as well as a refresh. The rows a statement's write does not keep are counted into its session and recorded by the accelerator write lock once the write completes; a failed write records nothing. A statement's row count reports rows inserted or replaced, as PostgreSQL's does. Cayenne's write already counts that way: a copy of a key that a later copy in the same INSERT replaced is not counted, and a row replacing a stored key is. Pin it in memory and file mode. * Fail an UPDATE that gives a row a key another row keeps An UPDATE that reassigned a Cayenne table's primary key ran as a delete and an insert, so a row moved onto a stored key silently replaced it, rows moved onto one new key collapsed to one row, and a NULL key deleted the row before the insert rejected it. Per #14576 such an UPDATE now fails and changes nothing, as in PostgreSQL: Failed to update dataset 'events': the new 'id' of 1 row matches a key already stored, so nothing was changed. UpdateExec materializes the updated rows once and checks those exact rows with the table's UpdateValidator before deleting anything; Cayenne's validator refuses a NULL new key, a key two updated rows share, and a key a row the statement leaves in place holds. A partitioned table prepares every partition's update before writing any, so a refusal in one partition changes none. * Describe the UPDATE key check and statement counting in the technical reference * Write an overwrite's position deletes into the snapshot it publishes An overwrite that repeats keys on a position-deletion table hid the superseded copies with deletion vectors written under the table's current snapshot, the one the overwrite replaces. That directory is then retired: its deletion vectors keep it alive, a failed overwrite (which removes only its own directory) strands them, and the wholesale reclaim that runs when a table has no snapshot manifest would remove deletion vectors the published snapshot still references. Write them under the snapshot the overwrite publishes. Through spiced, a file-mode full refresh repeating keys referenced four delete files under the previous snapshot directory; the new test fails before this change with "delete file .../<replaced>/deletions/... is outside the published snapshot". * Say when dataset_acceleration_refresh_rows_written and rows_superseded are counted rows_written counts rows as a refresh reads them, and rows_superseded once the refresh succeeds, so a refresh that fails is counted in the first only. The description no longer implies the two subtract to the rows kept. * Format the overwrite position-delete test * Satisfy pedantic lints in the UPDATE and buffered-version paths * Keep a read-write dataset's writes in the acceleration with write_mode: acceleration Any on_conflict value used to keep a read-write accelerated dataset's writes (and its transactions) in the acceleration, outside a change stream, and let its source be read-only. That routing was never documented and tied write placement to a conflict setting #14576 deprecates. A new `acceleration.write_mode: acceleration` selects it instead, and on_conflict no longer routes writes: - write_through (default) and write_back still need a source that accepts writes; acceleration does not. A sink source is unchanged. - A read-write dataset over a read-only source without it fails at load, naming the fix; write_mode: acceleration with refresh_mode: changes fails too, since the source's changes would overwrite the writes. - The on_conflict deprecation warning now reaches read-write datasets. With on_conflict routing removed, 24 integration tests over read-write file-sourced Cayenne datasets (DML, transactions, the UPDATE key check) failed to load; with write_mode: acceleration set they and the rest of the affected suites pass (54/54). * Build the partitioned UPDATE fixture without format_push_string * Refuse write_mode: acceleration with a connector's default change stream The check read the literal refresh_mode, so a cdc: dataset that omits it, and refreshes by changes by its connector's default, loaded with write_mode: acceleration and let the change stream overwrite its writes. It now reads the connector-resolved mode, and the message says the dataset refreshes by `changes` rather than quoting a setting it may not have. Through spiced, a cdc: dataset with write_mode: acceleration and no refresh_mode loaded before (count 0) and is refused after. write_mode_acceleration_is_refused_with_a_default_change_stream (needs the debezium feature) fails with the literal check ("got Some(Ready)") and passes with this one. * fix(runtime-table): keep the overlap dedupe for append refreshes that keep the last arrival An append refresh on a Cayenne table with a time column but no usable primary key skipped the exact-row dedupe, so every refresh re-appended its whole refresh_append_overlap window. * fix(runtime): seed superseded-row metrics from the connector-resolved refresh mode A connector whose default refresh mode is changes, with refresh_mode omitted, was seeded as full and published an arrival series that its refreshes never report. --------- Co-authored-by: Phillip LeBlanc <[email protected]> * fix(object-store): stop reading a response body at its first error (#14831) object_store's SpawnedReqwestConnector forwards response-body frames from its I/O task until the body ends. Once a request outlives client_timeout, reqwest's body returns a new timeout error on every poll and never ends, so the task pushes errors into an unbounded channel until object_store drops the body after its retry backoff and retry request. On a ~5 MB/s link, loading the 195 MiB TPC-H lineitem file from S3 grew RSS past 25 GB. Add object_store_spawn::SpawnedReqwestConnector, which runs request I/O on the given runtime the same way but stops after forwarding the first body error, use it at every call site, and disallow the upstream constructor. Fixes #14830 * fix(ci): use byte-order collation in the benchmark Postgres container (refs #14815) (#14836) * fix(listing): keep a partition predicate as a residual filter on the `_location` fast path (#14790) `MetadataPruningListingTable::scan` takes a fast path when the query carries a `_location` predicate: it opens exactly the named objects and applies only `_location` and `_last_modified`. It applies no other predicate. `supports_filters_pushdown`, however, only reported `Inexact` for the `_last_modified` prune, and explicitly fell through to the inner listing whenever a `_location` predicate was present. The inner `ListingTable` reports a partition-column predicate `Exact`, so DataFusion dropped it from the plan while the fast path never applied it. A query combining `_location` with a partition predicate then returned rows the predicate excludes. Report every filter `Inexact` whenever either path applies, so the residual `FilterExec` re-checks them while the prune still removes the files that cannot match. The four `_location` EXPLAIN snapshots are unaffected: each filters on `_location` alone, and for a lone `_location` filter the inner listing already answers `Inexact` (their committed plans carry a `Filter` above a `partial_filters` scan), which is what this now returns directly. Co-authored-by: grokspice <[email protected]> Co-authored-by: Emma <[email protected]> * fix(ci): search only the unixodbc keg, not all of Homebrew's lib, in the macOS release builds (#14846) `RUSTFLAGS=-L $brew_prefix/lib` is searched before a build script's own link-search path, so a static library Homebrew has linked into that directory under the same name replaces the vendored copy when rustc bundles it. On a runner holding a foreign libcrypto.a, openssl-sys bundles that instead of the vendored OpenSSL librdkafka was compiled against, and the spiced link fails on undefined _ENGINE_* symbols. The odbc build only needs libodbc, which the unixodbc keg's own lib directory provides; setup-cc already scopes every other macOS job the same way. * fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC (fixes #14753) (#14840) * fix(federation): keep date, timestamp and interval values local on SQLite reached through ADBC or ODBC The generic-driver routes installed no SQLite expression gate, so a timestamp literal federated as CAST('...' AS TIMESTAMP), which SQLite evaluates to the year as an integer (every row matched), and an interval as INTERVAL '1 HOURS', which SQLite cannot parse. The sqlite engine now gets the SQLite connector's gate plus a refusal of every temporal value. Fixes #14753 Fixes #14754 * test: simplify the SQLite temporal-gate regression tests * docs: say why the sqlite engine has a federation gate * test(odbc): the SQLite profile now keeps unfaithful casts and temporal values local * test(odbc): give the SQLite temporal checks their own test, with a date literal * fix(odbc): apply the federation policy to the scan's filter pushdown too ODBCTableFactory built its SqlTable without the function-support policy, so when the federation analyzer refused a plan, the filter fell back to the scan's own pushdown, which consulted no policy. On the ODBC SQLite profile a timestamp filter was pushed as CAST('...' AS TEXT) and returned every row. --------- Co-authored-by: t <[email protected]> * chore: pin the fork at spiceai/datafusion#249 and guard the DataFusion fixes backported from 54 (#14800) * build: pin the DataFusion 55 / Arrow 59 fork lines Moves every fork to its DataFusion 55 line: arrow-rs 59.3.0, DataFusion 55.1.0, Ballista (upstream main on DataFusion 55), federation, table-providers, iceberg-rust 0.10.1, duckdb-rs 1.4.4 on arrow 59, ADBC 24, snowflake-rs, spark-connect-rs, spice-rs and spicebench on arrow 59. datafusion-functions-json comes from crates.io (0.55.6); delta-kernel stays on 0.27.1 with its default engine split into delta_kernel_default_engine. docs/dev/fork_patches.md is re-audited against the new lines; the pins stay marked TEMPORARY until the fork pull requests merge. * feat: upgrade to DataFusion 55 and Arrow 59 - Statistics flow through DataFusion 55's StatisticsContext. The deprecated partition_statistics returns Absent for built-in plans on 55, so every wrapper implements child_stats_requests / statistics_from_inputs and the optimizer rules, join sizing and Flight batch sizing read real statistics again. - ExecutionPlan / TableProvider wrappers implement the new required and defaulted methods explicitly (apply_expressions, replace_children, try_to_proto, merge_into, ...), forwarding where they wrap. - Arrow 59 refuses a MAP whose entries field is nullable: Flight SQL reads replace the declared schema before decoding and normalize the map afterwards, and legacy IPC maps decode as a list (arrow_tools map_entries). - Delta Lake refuses Void and interval columns with a structured error instead of mis-reading them. - Ballista 55: Vcores, the protocol-version handshake, task-id based cancellation and the append-only task-info model. - Vortex 0.86 and the remaining API moves (TableReference, TableScanBuilder, TableSchema builder, codec proto converter, WriteOp::MergeInto). * ci: validate the datafusion-table-providers pin against spiceai-55-patches The DataFusion 55 line of the fork is spiceai-55-patches. * revert: drop the unfinished guard tests d65aa29 picked up by accident d65aa29 was meant to change only the CI branch check; it also committed work-in-progress test edits. Restore those files to 98800a85; the finished tests land in their own commit. * fix(duckdb): push regexp_count down NULL-preserving, as DataFusion 55 evaluates it DataFusion 55 (apache/datafusion#24239) makes regexp_count propagate NULL, as PostgreSQL does. The DuckDB rendering still wrapped the count in coalesce(.., 0) to match the old kernel, so a DuckDB-accelerated dataset answered 0 where local evaluation answers NULL, and a WHERE on the count kept rows local evaluation drops. Render it as len(regexp_extract_all(..)), which is NULL for a NULL input or pattern. * test(cayenne): count -0.0 as equal to 0.0 in the boundary-values test DataFusion 55 compares floats in SQL by IEEE 754 value, so float_val = 0.0 matches the -0.0 row too (4 rows, not 3). A MemTable in the same session returns the same rows, so this is the expectation, not Cayenne. * build: repin the forks and guard their new patches - Ballista 8bf37ad0: the executor poll loop accepts a vcore semaphore with no permits yet. Upstream's assert panicked the loop at startup, so no Spice executor registered and every cluster query failed. - DataFusion d98ae645 (MSRV-clean unparser), federation 9ca84a39, table-providers 661d16d1, iceberg 575d81a1: their own tests now run against the Spice DataFusion (and, for table-providers, ADBC) forks. - Guard tests for the unparser rescope, eager aggregation through StatisticsContext, SchemaCastScanExec statistics forwarding and Ballista TaskInfo fields 11/12; the ledger names them and records the Ballista fix. - The distributed EXPLAIN snapshot now reads Ballista 55's UnknownPartitioning(1) where it printed None. * build: point the fork pins at the spiceai-*-patches-2 branches The DataFusion 55 lines of arrow-rs, datafusion, datafusion-ballista, datafusion-federation, datafusion-table-providers, duckdb-rs, snowflake-rs and spark-connect-rs are reviewed on new spiceai-*-patches-2 branches, leaving the earlier spiceai-*-patches branches untouched. Same revisions; only the branch names in the pin comments, the fork-patch ledger and the table-providers pin check change. * fix: re-root a delegated projection on the inner plan before swapping DataFusion 55's projection pushdown reads a projection's input as the node being swapped with: `ProjectionExec` collapses the chain starting at `projection.input()`, `CoalescePartitionsExec` rebuilds from its child. A wrapper that handed its inner plan the projection unchanged (its input being the wrapper) got it back still above the wrapper, and the pushdown then nested another copy on every step: the Cayenne maintained-aggregate soundness test grew to gigabytes and never finished. CayenneAccelerationExec, the DuckDB aggregate pushdown marker and PartitionedUnionExec now rebuild the projection over their inner plan (whose schema they share) before delegating. Regression tests cover the Cayenne wrapper and the DuckDB marker. * fix(cayenne): read legacy inline map data under Arrow 59 and keep the type policy - Inline data written under a nullable map-entries declaration is read through arrow_tools::map_entries, since Arrow 59 refuses that declaration at decode. - The primary-key fast-path test's control query no longer uses a bare SUM, which DataFusion 55 answers from statistics without a scan. - RunEndEncoded columns stay refused: the Vortex pin can store them, but accepting them is a product change, so the drift test records the exception. * fix: bring the remaining tests and connectors in line with DataFusion 55 and Arrow 59 - Flight: repair a server's nullable map-entries declaration before decoding (Arrow 59 refuses it), as the Flight SQL path does. - Federation: SQLite and MySQL refuse a filter on a volatile projection output (DataFusion fork unparser), Turso forwards it, MSSQL opts out. - Declared types: parse the decimal form directly so an out-of-range precision/scale still names the problem under arrow-rs's stricter parser. - Expectations that DataFusion 55 changed: predicate reordering (reorder_q8), binary literals rendered as X'..', sorts moved below projections (vector scans), the NSQL context's version and Spark function list, and the null-aware anti join corrected in Ballista's distributed planner. - json_get tests follow datafusion-functions-json 0.55 (negative array indices, json_as_text for a string cast, integral floats in json_get_int). * fix(vortex): port the merged point-lookup and key-block code to DataFusion 55 and Vortex 0.86 Trunk's point-lookup and key-block changes were written against DataFusion 54 and the earlier Vortex pin. The scan now binds its projection and filter before handing them to Vortex (the reader takes bound expressions), the point read keeps trunk's optimize-then-bind of its conjuncts, dynamic filters are detected through DynamicFilterTracking, and TableSchema is built with From. Two merged tests move to the current Vortex Arrow conversion and DataFusion's non-zero batch_size. * fix(arrow_tools): keep the producer's dictionary ids when repairing a map declaration Repairing a nullable map-entries declaration re-encoded the IPC schema, and re-encoding renumbers dictionaries from zero while the dictionary batches that follow keep the producer's ids. A producer that numbers them differently then failed to decode, or decoded each dictionary column against another's dictionary and returned wrong values with no error (reproduced on the Flight SQL read, the Flight read and the Flight subscribe paths). The repair now patches the schema message in place: one byte per map field (the entries nullability, or the Map-to-List type tag for the decodable form), verified with the flatbuffers verifier and checked against the expected schema; an unexpected layout is refused rather than passed through. * fix(runtime): keep the built-in semantics of functions datafusion-spark 55 would shadow datafusion-spark 55 registers Spark versions of power/pow, atan2 and concat_ws over the built-ins (infinity instead of an error for zero to a negative power, Float64 widening, array flattening) and adds hypot, monthname, quote and weekday. None of that is a deliberate change, so those functions join trunc, date_trunc and date_part in the withheld list, and a test pins exactly which names resolve to a Spark implementation. The NSQL context now lists the functions the session actually registers, so it no longer advertises the withheld ones. * fix(duckdb): keep a binary literal out of SQL sent to DuckDB DataFusion 55's unparser renders a binary literal as X'ff', which DuckDB 1.4.4 reads as the text 'xff': through DuckDB federation, b = X'ff' returned the row holding the bytes 'xff' instead of 0xFF (and X'' missed the empty blob), with no error. Expressions with a binary literal now stay local; a NULL still pushes down. Covered end to end by duckdb_binary_literal_filters_agree_with_local_evaluation. * build: pin DataFusion at 006f3d21, which drops input sums from join statistics DataFusion 55 answers a bare SUM from exact statistics, and inner/outer join statistics carried each input's sum through, so SUM over a join returned the whole input table's sum (silent wrong results; caught by the Cayenne result-correctness parity tests). The fork fix is spiceai/datafusion#239; the ledger records it with those parity tests as its guard. * test(runtime-datafusion): drop a redundant clone in the binary-literal test * test: fix the Spark built-ins table width and report the pinned DataFusion revision - the_built_session_keeps_the_built_in_math_and_string_functions: the expected table padded one column a character wider than the rendered output; the values were already right. - spice-substrait-compliance reports DATAFUSION_FORK_REV, which still named the pre-upgrade revision; it now matches the workspace pin, as its test requires. * test(oracle): re-record the pushdown plan for DataFusion 55's decimal literal display DataFusion 55 prints a decimal literal as Decimal128(123.45,10,2) where 54 printed Decimal128(Some(12345),10,2). The SQL sent to Oracle is unchanged. * build: repin the forks to their merged fix PRs and carry spiceai-54 forward - DataFusion f22d1a74: spiceai-54 forward-merged into the 55 line (spiceai/datafusion#241), bringing #237 (a Date32 literal's cast uses the dialect's date type; SQLite read CAST('…' AS DATE) as a number, so date ranges pushed to SQLite matched nothing) and metadata-column listing pruning. - arrow-rs be4918e7, Ballista 7c4d54c2, table-providers 3045960a, iceberg 50e982dc, arrow-adbc 674d6184: the merged CI fix PRs. - Guard for #237: a Date32 range unparsed for SQLite keeps the rows it selects (runs the SQL on SQLite with --features sqlite); it fails on the previous pin. - Ledger rows for both; the substrait harness reports the new DataFusion pin. * build: pin DataFusion at ae719330, the merge of spiceai/datafusion#241 Same tree as f22d1a74 (spiceai-54 forward-merged into the 55 line); now the head of spiceai-55-patches-2 rather than a temporary branch. * style: rustfmt the SQLite date-range federation test * build: pin arrow-rs at its spiceai-59 merge and DataFusion at spiceai-55-patches-3 - arrow-rs: 2e2cc330, the merge of spiceai/arrow-rs#29 into spiceai-59 (same tree as the previous pin). - DataFusion: fc84f2d1 on spiceai-55-patches-3, which carries the Date32 literal fix (spiceai/datafusion#237) but not the metadata-column listing pruning (#229); the ledger records that pruning as not carried on 55 until it has a guard here. - Correct the spark-connect-rs branch annotation to spiceai-59-patches-2. * build: pin DataFusion at the spiceai-55 merge, and iceberg-rust and snowflake-rs at their arrow-rs bumps - DataFusion: 02550cf9, the merge of spiceai/datafusion#243 into spiceai-55 (same tree as the previous pin); no longer a temporary pin. - iceberg-rust: 0284be3d and snowflake-rs: b95a74e8, which move each fork's own arrow-rs pin to 2e2cc330 (Cargo manifests only; Spice builds the same code). * build: repin ballista, table-providers, iceberg-rust and spark-connect-rs to their branch heads - datafusion-ballista 557ae4ea: arrow-rs/DataFusion landed pins, and the Rust 1.99 clippy fixes (#70, #71, #72). - datafusion-table-providers 5f50cfd5 and iceberg-rust a0bd6b04: arrow-rs/DataFusion landed pins (#82, #53). - spark-connect-rs 3bf6421a: regenerated lockfile. datafusion-federation and duckdb-rs stay at 9ca84a39 and 8ee43073: table-providers still pins exactly those revisions, and a second revision of either would put two copies of the crate in the graph. Their newer heads only change their own Cargo manifests and lockfile. * build: pin iceberg-rust, snowflake-rs, spark-connect-rs, spice-rs and spicebench at their merges The fork PRs these pins pointed at have merged, mostly as squash merges, so several pinned commits are no longer on their base branches. Each pin now points at the merge on the fork's base branch: - iceberg-rust spiceai-0.10.1-df-55 @ 3e2a14e (spiceai/iceberg-rust#50, #54) - snowflake-rs spiceai-59 @ f555738 (spiceai/snowflake-rs#13, #15) - spark-connect-rs spiceai-59-2 @ 18ae9bd (spiceai/spark-connect-rs#15, #16) - spice-rs trunk @ 4429f39 (spiceai/spice-rs#103) - spicebench trunk @ 8a90555 (spiceai/spicebench#284) datafusion-federation stays at 9ca84a3, which is on spiceai-55 because spiceai/datafusion-federation#88 was a merge commit; only its branch note changes. What changes for Spice: lint-only rewrites in snowflake-api, and tests in iceberg-datafusion and spark-connect-core. The other differences are in the forks' own [patch.crates-io] sections, which cargo ignores for a dependency. spice-rs's tree is identical. * build: pin the forks at their landed version-branch revisions - arrow-adbc 6e4119ac (spiceai-24, #7), iceberg-rust 3e2a14e8 (spiceai-0.10.1-df-55, #50), datafusion-table-providers 465926a3 (spiceai-55, #80), snowflake-rs f5557381 (spiceai-59, #13), spark-connect-rs 18ae9bd3 (spiceai-59-2, #15), spice-rs 4429f395 and spicebench 8a90555d (trunk). - datafusion-federation stays at 9ca84a39, now on spiceai-55 (#88): datafusion-table-providers pins exactly that revision. - duckdb-rs stays at 8ee43073 and temporary: #49 was squash-merged, so that revision is not on spiceai-1.4.4, and datafusion-table-providers pins it, so both have to move together. - datafusion-ballista is unchanged; its version PR is still open. * build: pin datafusion-ballista at upstream 55.0.0-rc1 spiceai/datafusion-ballista#73 merged upstream's 55.0.0-rc1 tag into spiceai-55-patches-2, the head of spiceai/datafusion-ballista#68. #68 is still open, waiting on license-compliance decisions. Pin 718429f, the #73 merge, until #68 lands on spiceai-55. ballista-core, -executor, -scheduler, -history and -api-types move from 54.0.0 to 55.0.0; no packages are added or removed in Cargo.lock. `cargo check -p spiced --locked` passes on rustc 1.98.1. * docs(fork_patches): record the datafusion-ballista pin at 718429f4 (upstream 55.0.0-rc1, #73) * build: pin datafusion-ballista at a7c4c585, the merge of spiceai/datafusion-ballista#68 into spiceai-55 * build: pin datafusion-ballista at the spiceai-55 merge of #68 spiceai/datafusion-ballista#68 merged into spiceai-55 as a7c4c58, a merge commit whose tree is identical to 7184…

kosiew and others added 30 commits July 25, 2026 10:39

Copilot AI reviewed Oct 2, 2026

View reviewed changes

build: pin arrow-rs at 2e2cc330, the merge of spiceai/arrow-rs#29 int…

f5b39dc

…o spiceai-59 (#244)

Copilot AI balanced review requested due to automatic review settings October 2, 2026 01:29

Copilot started reviewing on behalf of krinart October 2, 2026 01:29

View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Comment thread Cargo.toml Resolved

Jeadie approved these changes Oct 2, 2026

View reviewed changes

krinart marked this pull request as draft October 2, 2026 02:09

phillipleblanc approved these changes Oct 2, 2026

View reviewed changes

krinart marked this pull request as ready for review October 2, 2026 02:33

phillipleblanc merged commit 02550cf into spiceai-55 Oct 2, 2026
2 checks passed

krinart added a commit to spiceai/datafusion-ballista that referenced this pull request Oct 2, 2026

build: pin DataFusion at 02550cf9, the merge of spiceai/datafusion#243 …

1370715

…into spiceai-55

krinart mentioned this pull request Oct 2, 2026

Pin arrow-rs and DataFusion at their landed spiceai merges spiceai/datafusion-ballista#70

Merged

krinart added a commit to spiceai/datafusion-federation that referenced this pull request Oct 2, 2026

build: pin DataFusion at 02550cf9, the merge of spiceai/datafusion#243 …

579714a

…into spiceai-55

krinart mentioned this pull request Oct 2, 2026

Pin arrow-rs and DataFusion at their landed spiceai merges spiceai/datafusion-federation#89

Merged

krinart added a commit to spiceai/datafusion-table-providers that referenced this pull request Oct 2, 2026

build: pin DataFusion at 02550cf9, the merge of spiceai/datafusion#243 …

9377e24

…into spiceai-55

This was referenced Oct 2, 2026

Pin arrow-rs and DataFusion at their landed spiceai merges spiceai/datafusion-table-providers#82

Merged

Pin DataFusion at the spiceai-55 merge (02550cf9) spiceai/iceberg-rust#53

Merged

krinart added a commit to spiceai/iceberg-rust that referenced this pull request Oct 2, 2026

build: pin DataFusion at 02550cf9, the merge of spiceai/datafusion#243 …

a0bd6b0

…into spiceai-55 (#53)

lukekim mentioned this pull request Oct 3, 2026

chore: merge upstream ADBC 24 into spiceai-24 spiceai/arrow-adbc#7

Merged

lukekim pushed a commit to spiceai/iceberg-rust that referenced this pull request Oct 3, 2026

build: pin DataFusion at 02550cf9, the merge of spiceai/datafusion#243 …

e530b39

…into spiceai-55 (#53)

krinart mentioned this pull request Oct 3, 2026

Upgrade to DataFusion 55.1 and Arrow 59.3 spiceai/spiceai#14612

Merged

Navigation Menu

Repository navigation

Merge upstream DataFusion 55.1.0 into spiceai-55 - #243

Merge upstream DataFusion 55.1.0 into spiceai-55#243
phillipleblanc merged 901 commits into
spiceai-55from
spiceai-55-patches-3

krinart commented Oct 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

krinart commented Oct 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants