Skip to content

Commit 1f21115

Browse files
committed
pyarrow 26.0.0: the matrix and the Python suites run on it (venv, CI pin). Four pyarrow bugs the matrix worked around on 25.0.1 are fixed in 26.0.0 (utf8_normalize composing for NFC/NFKC by our apache/arrow#51237; pairwise_diff, the boolean fill-null kernels and winsorize honouring the offset): their pinning tests failed as designed and are retired with the _materialised copies and the unicodedata oracle; binary_slice's default-stop overflow remains (apache/arrow#51222) with its test. Matrix on 26.0.0: 38,898 pass, 1,643 documented, 0 unclassified (winsorize/float64 gained: it was the offset bug, not finding 22). Three more pinned behaviours changed and their tests say so (cumulative_max default start, out-of-range float-to-int cast now wraps as the GPU does, the fixed-size-list Parquet file pyarrow's reader used to reject); the registry carries 26.0.0's five new names: remainder on the GPU as modulo() has always computed it, remainder_checked, floored modulo/modulo_checked and search_sorted recorded as missing; the matrix's modulo checks against pc.remainder. UPSTREAM rows read fixed in 26.0.0; EVALUATION, COVERAGE, ARROW_FUNCTIONS, differential_coverage.json regenerated; FINDINGS Round 22; JSON/CSV differences re-checked on 26.0.0
1 parent 51f2e30 commit 1f21115

16 files changed

Lines changed: 211 additions & 189 deletions

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ jobs:
3030
with:
3131
python-version: "3.13"
3232
- name: Python packages
33-
run: python -m pip install pyarrow==25.0.1 polars==1.44.1 numpy==2.5.3 pytest==9.1.1
33+
run: python -m pip install pyarrow==26.0.0 polars==1.44.1 numpy==2.5.3 pytest==9.1.1
3434
- name: Conformance subset
3535
env:
3636
PY: python

‎CHANGELOG.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,17 @@
22

33
## Unreleased
44

5-
Nothing yet.
5+
- The differential matrix and the Python test suites run on pyarrow 26.0.0 (CI pin, venv). Four pyarrow
6+
bugs the matrix worked around on 25.0.1 are fixed in 26.0.0 (`utf8_normalize` composing for NFC and
7+
NFKC, by our apache/arrow#51237; `pairwise_diff`, the boolean fill-null kernels and `winsorize` on a
8+
sliced array honouring the offset): their pinning tests failed on 26.0.0 as designed and were retired
9+
with the workarounds, so the oracle is `pyarrow.compute` on the slice itself again. The `binary_slice`
10+
default-stop overflow remains (apache/arrow#51222), with its pinning test. Three more pinned behaviours
11+
changed in 26.0.0 and their tests say so now (`cumulative_max`'s default start, the out-of-range
12+
float-to-int cast, the fixed-size-list Parquet file pyarrow's reader used to reject), and the registry
13+
carries 26.0.0's five new function names: `remainder` runs on the GPU as `modulo()` has always computed
14+
it (the C `%`), `remainder_checked`, the floored `modulo` / `modulo_checked` and `search_sorted` are
15+
recorded as missing.
616

717
## 0.5.0 — 2026-10-08
818

‎docs/ARROW_FUNCTIONS.md‎

Lines changed: 27 additions & 22 deletions
Large diffs are not rendered by default.

‎docs/COVERAGE.md‎

Lines changed: 27 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ a claim can be checked in one jump.
1212

1313
This file groups Arrow's functions into families and explains each one. The companion page
1414
[**ARROW_FUNCTIONS.md**](ARROW_FUNCTIONS.md) does the complementary thing: one row per **exact Arrow
15-
function name**, all 307 of them, generated from a registry that the test suite executes name by name
15+
function name**, all 312 of them (Arrow v26; 307 in v25), generated from a registry that the test suite executes name by name
1616
against `pyarrow.compute`. Go there for "is `<name>` covered?"; stay here for "how does this family work?".
1717

1818
## Summary
@@ -46,7 +46,9 @@ aggregations section below is likewise outside the compute-function total.
4646

4747
A row here covers a family, so these are not function counts. The by-name numbers are in
4848
[ARROW_FUNCTIONS.md](ARROW_FUNCTIONS.md): of Arrow v25's 307 compute function names, **all 307 are
49-
reachable** — 283 entirely on the GPU, 17 on the host, 7 with a stated limitation, and none missing.
49+
reachable** — 284 entirely on the GPU, 17 on the host, 7 with a stated limitation, and none missing. Of
50+
the five names Arrow v26 added, `remainder` runs on the GPU (it is `modulo()`'s C `%`) and the floored
51+
`modulo`, the two checked variants and `search_sorted` are recorded as missing.
5052

5153
**The scope ArrowMetal 0.4.0 answers to, name by name:** every Arrow compute function name — 7 of them
5254
with a stated limitation — over `int8/16/32/64`, `uint8/16/32/64`, `float16/32/64`, `bool`, `utf8`,
@@ -201,9 +203,10 @@ exactly what the atomic path could not: Float64 sums and means (through the soft
201203

202204
## Element-wise arithmetic
203205

204-
`modulo` (`%`, C remainder semantics, `x % 0` defined as 0) is an ArrowMetal extension rather than an
205-
Arrow function name, so it has no row of its own; it lives beside `power` in `Kernels/Rounding.swift`
206-
and is reachable as `am_binary(op 5)` and `.modulo()` / `__mod__` in Python.
206+
`modulo` (`%`, C remainder semantics, `x % 0` defined as 0) lives beside `power` in `Kernels/Rounding.swift`
207+
and is reachable as `am_binary(op 5)` and `.modulo()` / `__mod__` in Python. Since Arrow 26.0.0 it has a
208+
function name of its own, `remainder` (the C `%`); Arrow's floored `modulo`, the checked pair and
209+
`search_sorted`, new in 26.0.0, are not implemented and carry `missing` rows in the registry.
207210

208211
| Arrow function | Status | Notes |
209212
|---|---|---|
@@ -448,11 +451,11 @@ grouped aggregates, covered above. The row below is an ArrowMetal extension.
448451

449452
<!-- BEGIN differential cases per function: generated by python/tests/coverage_report.py -->
450453

451-
The differential matrix (`python/tests/test_differential.py`, [TESTING.md](TESTING.md#3-the-differential-matrix)) runs 217 operations over 46 column types: 1,544 (operation, type) cells, each over 27 datasets, 41,688 cases. The run recorded in `docs/data/differential_coverage.json` (ArrowMetal 0.5.0, pyarrow 25.0.1, Apple M4 Max, router `gpu`): **38,897 pass, 1,644 documented divergences, 1,147 skipped, 0 unclassified**.
454+
The differential matrix (`python/tests/test_differential.py`, [TESTING.md](TESTING.md#3-the-differential-matrix)) runs 217 operations over 46 column types: 1,544 (operation, type) cells, each over 27 datasets, 41,688 cases. The run recorded in `docs/data/differential_coverage.json` (ArrowMetal 0.5.0, pyarrow 26.0.0, Apple M4 Max, router `gpu`): **38,898 pass, 1,643 documented divergences, 1,147 skipped, 0 unclassified**.
452455

453456
A case counts toward an Arrow function when the ArrowMetal call it runs is that function's implementation (the call column of [ARROW_FUNCTIONS.md](ARROW_FUNCTIONS.md)) or, for an ArrowMetal call with no Arrow name of its own (`sort`, `argsort`, `to_strings`), when its answer is compared with that function's result. One case can check several functions — a `compare_scalar` case runs all six comparisons — so the rows below add up to more than the case total. A documented divergence is a failing case that matches an open finding in [EVALUATION.md](EVALUATION.md); a skipped case is an operation that does not implement that column type.
454457

455-
280 of the 307 Arrow function names have matrix cases. Every name, with or without them, is also run through `call_function` against `pyarrow.compute` in `python/tests/test_functions.py`.
458+
281 of the 312 Arrow function names have matrix cases. Every name, with or without them, is also run through `call_function` against `pyarrow.compute` in `python/tests/test_functions.py`.
456459

457460
| Arrow function | Section | Matrix operations | Cases | Pass | Documented | Skipped |
458461
|---|---|---|---:|---:|---:|---:|
@@ -491,12 +494,23 @@ A case counts toward an Arrow function when the ArrowMetal call it runs is that
491494
| `add_checked` | Arithmetic | `arith_checked` | 270 | 266 | 4 | 0 |
492495
| `divide_checked` | Arithmetic | `arith_checked` | 270 | 266 | 4 | 0 |
493496
| `multiply_checked` | Arithmetic | `arith_checked` | 270 | 266 | 4 | 0 |
497+
| `remainder` | Arithmetic | `modulo` | 216 | 216 | 0 | 0 |
494498
| `negate_checked` | Arithmetic | `unary_checked` | 270 | 270 | 0 | 0 |
495499
| `power_checked` | Arithmetic | `power_checked` | 216 | 216 | 0 | 0 |
496500
| `sqrt_checked` | Arithmetic | `log_checked` | 54 | 54 | 0 | 0 |
497501
| `subtract_checked` | Arithmetic | `arith_checked` | 270 | 266 | 4 | 0 |
498502
| `expm1` | Arithmetic | `math_extra` | 54 | 46 | 8 | 0 |
499503
| `hypot` | Arithmetic | `math_extra` | 54 | 46 | 8 | 0 |
504+
| `array_sort_indices` | Sorts | `argsort`, `argsort_nulls_first`, `argsort_total_order`, `sort`, `sort_indices`, `top_k`, `top_k_nulls_first` | 1,944 | 1,944 | 0 | 0 |
505+
| `sort_indices` | Sorts | `lexsort`, `sort_indices` | 594 | 594 | 0 | 0 |
506+
| `partition_nth_indices` | Sorts | `partition_nth_indices` | 270 | 270 | 0 | 0 |
507+
| `select_k_unstable` | Sorts | `select_k_unstable` | 270 | 270 | 0 | 0 |
508+
| `top_k_unstable` | Sorts | `select_k_unstable` | 270 | 270 | 0 | 0 |
509+
| `bottom_k_unstable` | Sorts | `select_k_unstable` | 270 | 270 | 0 | 0 |
510+
| `rank` | Sorts | `ranking` | 270 | 270 | 0 | 0 |
511+
| `rank_quantile` | Sorts | `rank_quantile_and_normal` | 270 | 270 | 0 | 0 |
512+
| `rank_normal` | Sorts | `rank_quantile_and_normal` | 270 | 270 | 0 | 0 |
513+
| `winsorize` | Sorts | `winsorize` | 270 | 268 | 2 | 0 |
500514
| `bit_wise_and` | Bitwise | `bitwise_and` | 216 | 216 | 0 | 0 |
501515
| `bit_wise_not` | Bitwise | `bitwise_not` | 216 | 216 | 0 | 0 |
502516
| `bit_wise_or` | Bitwise | `bitwise_or` | 216 | 216 | 0 | 0 |
@@ -700,16 +714,6 @@ A case counts toward an Arrow function when the ArrowMetal call it runs is that
700714
| `drop_null` | Selections | `drop_null` | 351 | 297 | 0 | 54 |
701715
| `inverse_permutation` | Selections | `permutation` | 270 | 270 | 0 | 0 |
702716
| `scatter` | Selections | `permutation`, `scatter` | 621 | 621 | 0 | 0 |
703-
| `array_sort_indices` | Sorts | `argsort`, `argsort_nulls_first`, `argsort_total_order`, `sort`, `sort_indices`, `top_k`, `top_k_nulls_first` | 1,944 | 1,944 | 0 | 0 |
704-
| `sort_indices` | Sorts | `lexsort`, `sort_indices` | 594 | 594 | 0 | 0 |
705-
| `partition_nth_indices` | Sorts | `partition_nth_indices` | 270 | 270 | 0 | 0 |
706-
| `select_k_unstable` | Sorts | `select_k_unstable` | 270 | 270 | 0 | 0 |
707-
| `top_k_unstable` | Sorts | `select_k_unstable` | 270 | 270 | 0 | 0 |
708-
| `bottom_k_unstable` | Sorts | `select_k_unstable` | 270 | 270 | 0 | 0 |
709-
| `rank` | Sorts | `ranking` | 270 | 270 | 0 | 0 |
710-
| `rank_quantile` | Sorts | `rank_quantile_and_normal` | 270 | 270 | 0 | 0 |
711-
| `rank_normal` | Sorts | `rank_quantile_and_normal` | 270 | 270 | 0 | 0 |
712-
| `winsorize` | Sorts | `winsorize` | 270 | 267 | 3 | 0 |
713717
| `fill_null` | NullFilling | `fill_null` | 351 | 297 | 0 | 54 |
714718
| `fill_null_forward` | NullFilling | `fill_null_direction` | 297 | 297 | 0 | 0 |
715719
| `fill_null_backward` | NullFilling | `fill_null_direction` | 297 | 297 | 0 | 0 |
@@ -737,13 +741,12 @@ A case counts toward an Arrow function when the ArrowMetal call it runs is that
737741
| `hash_min` | GroupedAggregations | `group_by_min` | 270 | 189 | 0 | 81 |
738742
| `hash_max` | GroupedAggregations | `group_by_max` | 270 | 189 | 0 | 81 |
739743

740-
The 17 operations that check no Arrow compute function, with what they check:
744+
The 16 operations that check no Arrow compute function, with what they check:
741745

742746
| Matrix operation | What it checks | Cases | Pass | Documented | Skipped |
743747
|---|---|---:|---:|---:|---:|
744748
| `extension_type` | extension name, metadata and storage through the C Data Interface | 216 | 216 | 0 | 0 |
745749
| `hash64` | ArrowMetal's 64-bit hash: null positions kept, equal values hash equal, no collisions | 324 | 324 | 0 | 0 |
746-
| `modulo` | integer remainder, against Arrow's `divide`, `multiply` and `subtract` (Arrow has no modulo) | 216 | 216 | 0 | 0 |
747750
| `months_between` | whole months between two dates, against Arrow's `year` and `month` arithmetic | 270 | 224 | 46 | 0 |
748751
| `nulls_constructor` | `am.nulls(n)` against `pa.nulls(n)` | 27 | 27 | 0 | 0 |
749752
| `percent_rank_and_cume_dist` | window functions, against a reference written in the harness | 270 | 270 | 0 | 0 |
@@ -879,7 +882,7 @@ included.
879882

880883
## What ArrowMetal 0.4.0 claims, and what it does not
881884

882-
**The claim.** ArrowMetal 0.4.0 answers to **all 307 of the Apache Arrow v25 compute function names** —
885+
**The claim.** ArrowMetal 0.5.0 answers to **all 307 of the Apache Arrow v25 compute function names** (and to `remainder` of the five v26 added; the other four are recorded as missing) —
883886
the 283 in the C++ docs plus the 24 `hash_*` grouped aggregates — over `int8/16/32/64`, `uint8/16/32/64`,
884887
`float16/32/64`, `bool`, `utf8`, `binary`, `fixed_size_binary`, `decimal32/64/128`, `date32/64`,
885888
`time32/64`, `timestamp`, `duration`, the three `interval` layouts, `list` / `large_list` /
@@ -928,10 +931,11 @@ transforms; nested access; and Arrow C Data, C Device and C Stream interop for a
928931
`hash_distinct` rather than first-appearance order and a deterministic but not first-seen group
929932
order; `unique` / `value_counts` over a **utf8** column dropping the null (they keep it on every
930933
other type, as Arrow does, and their order option now defaults to Arrow's); an out-of-range
931-
`safe=false` float -> integer cast saturating at 64 bits rather than at the target's width — C
932-
leaves it undefined and `safe=true` refuses the row instead; `cumulative_*` carrying the running
934+
`safe=false` float -> integer cast wrapping rather than saturating at the target's width (pyarrow
935+
26.0.0 wraps the same way on the values the test pins; 25.0.1 saturated) — C leaves it undefined and
936+
`safe=true` refuses the row instead; `cumulative_*` carrying the running
933937
value across nulls; exact `approximate_median` where Arrow sketches; and NFC / NFKC composing, where
934-
`pyarrow.compute.utf8_normalize` does not. Two Arrow quirks are reproduced rather than corrected —
938+
`pyarrow.compute.utf8_normalize` did not until 26.0.0. Two Arrow quirks are reproduced rather than corrected —
935939
`ceil` on a calendar unit always advancing a boundary value, and rounding to a unit finer than the
936940
column's resolution flooring whatever mode was asked for — and one is not: Arrow's
937941
`floor_temporal(unit="week", multiple>1, calendar_based_origin=True)` can exceed its own input, and

‎docs/CSV.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ parse, bit for bit, by every float case in `test_csv.py`.
190190
## Matching pyarrow
191191
192192
pyarrow's rules were established by probing pyarrow 25.0.1 (the probe below), and every row of this table
193-
is a case in `python/tests/test_csv.py`.
193+
is a case in `python/tests/test_csv.py`; the cases pass unchanged on pyarrow 26.0.0 (run on 2026-10-15).
194194
195195
| Rule | pyarrow 25.0.1, matched |
196196
|---|---|

0 commit comments

Comments
 (0)