Repository navigation
perf(arrow-arith): vectorize checked integer add, sub and mul - #12058
Open
mbutrovich wants to merge 6 commits into
Open
mbutrovich wants to merge 6 commits into
mbutrovich wants to merge 6 commits into
Conversation
Contributor
Author
|
run benchmark arithmetic_kernels |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-r578ggp2-e53cf964-fxr7 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-c569p8q0-736ab66c-rp2l Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1bfdro2s-3785b8c4-c2s7 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (e8dc19e) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C target-cpu=x86-64-v3
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (e8dc19e) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C target-cpu=x86-64-v3 -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (9aed4ff) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1bfdro2s-44db1d15-szrn Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (e8dc19e) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C target-cpu=x86-64-v3
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1bfdro2s-3785b8c4-c2s7 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (e8dc19e) to af931c5 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: af931c5b0b
env:
BENCH_FILTER: ^int(32|64)_
RUSTFLAGS: -C target-cpu=x86-64-v3 -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1bfdro2s-44db1d15-szrn Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
github-actions
Bot
added
arrow-core
mbutrovich
marked this pull request as ready for review
November 3, 2026 19:32
mbutrovich
marked this pull request as draft
November 3, 2026 19:37
mbutrovich
marked this pull request as ready for review
November 4, 2026 00:30
mbutrovich
marked this pull request as draft
November 7, 2026 15:11
mbutrovich
marked this pull request as ready for review
November 12, 2026 23:24
sunchao
reviewed
Nov 13, 2026
sunchao
left a comment
sunchao
left a comment
Member
There was a problem hiding this comment.
Reviewed 9a685cfe. The fork passed the arrow-arith debug and release tests. I left inline comments on fallible output allocation, replay for nullable unsigned subtraction, and coverage of the overflow reduction.
| overflow |= overflowed; | ||
| value | ||
| }) | ||
| .collect(); |
Member
There was a problem hiding this comment.
Could this reserve the output buffer fallibly? For two nonempty arrays with no nulls, the current checked kernel maps allocation failure to ArrowError::MemoryError as part of #11912. If the allocator rejects the output allocation here, the infallible Vec collection can abort the process before the checked fallback runs. A fallible reservation in the fast path would preserve the caller's ability to handle that error.
Contributor
Author
There was a problem hiding this comment.
Could this reserve the output buffer fallibly?
Yes. collect_overflowing now reserves with Vec::try_reserve_exact and returns ArrowError::MemoryError on failure, as try_binary_no_nulls does. That also makes the nullable and scalar paths fallible. The checked kernel still aborts there, through BufferBuilder::new in try_binary and vec! in PrimitiveArray::try_unary. The null-slot check that runs after a flagged pass allocates two bitmaps through BooleanBuffer::collect_bool, which aborts on failure. It runs only when the result has nulls, where try_binary allocates infallibly too.
| }, | ||
| Op::SubWrapping => op!(l, l_s, r, r_s, l.sub_wrapping(r)), | ||
| Op::Sub => try_op!(l, l_s, r, r_s, l.sub_checked(r)), | ||
| Op::Sub => match overflowing_op(l, l_s, r, r_s, T::Native::sub_overflowing) { |
Member
There was a problem hiding this comment.
This forces a second pass for ordinary nullable unsigned subtraction. For example, UInt64 [10, null, 20] - 1 has a zero payload at the null row when built through PrimitiveBuilder. The first pass sees 0u64 - 1, fills and discards its output, and the checked kernel allocates again to return [9, null, 19]. This repeats on every batch with a null even when all valid rows succeed. Could you add a nullable UInt32 or UInt64 array-minus-scalar benchmark and avoid replay on this case? The current benchmark only exercises signed types.
Contributor
Author
There was a problem hiding this comment.
Could you add a nullable
UInt32orUInt64array-minus-scalar benchmark and avoid replay on this case?
Both done. When a slot overflowed and the result has nulls, overflowing_op now recomputes the overflow of every row into a bitmap with BooleanBuffer::collect_bool and masks it with the validity bits. If no valid row overflowed, it returns the values from the first pass and doesn't call the checked kernel. test_checked_integer_slice_null_scalar_and_lengths checks your example, including that the builder wrote 0 into the null slot.
uint32_subtract_checked_scalar(0.1) and uint64_subtract_checked_scalar(0.1) measure it. uint32_subtract_checked(0.1) and uint64_subtract_checked(0.1) take the same path, because the left array's null slots hold 0 too. The uint32 and uint64 cases take their left operands from a higher range than their right operands, so valid rows never underflow. Against the bench commit, uint32_subtract_checked_scalar(0.1) measured 5.92x on Neoverse-V2, 1.17x on Xeon 8581C, and 2.20x with x86-64-v3. uint64_subtract_checked_scalar(0.1) measured 4.59x, 1.13x, and 2.31x.
| // for strides up to 64 rows. | ||
| for len in 1..=130 { | ||
| let mut values = vec![1; len]; | ||
| values[len - 1] = i32::MAX; |
Member
There was a problem hiding this comment.
Could the overflow position vary independently of the batch length? With a full-vector loop followed by a scalar tail, len - 1 lands in the last lane of a full vector or in the tail. A missing overflow flag in an interior lane can pass this test. A fixed batch above the vectorization threshold with one overflowing row moved through the batch would cover that case, including the separate array–array loop. For the one-row edge checks, the current null mask also hides every signed RHS MAX because the signed edge list has 12 entries and MAX is at index 11. Sweeping the pairs without the mask would cover those cases.
Contributor
Author
There was a problem hiding this comment.
Could the overflow position vary independently of the batch length?
Yes. test_checked_integer_overflow_positions now moves one overflowing row through a 300-row batch for every integer type. It covers add, sub, and mul with the batch against an array and against a scalar, on either side, with that row valid and with it null. 300 rows cover two iterations of a 128-row loop (32 i8 lanes of AVX2, unrolled four times) plus a remainder. It's skipped under Miri, which doesn't run the vectorized code. With collect_overflowing changed to drop the flag on odd rows, it fails at row 1.
For the one-row edge checks, the current null mask also hides every signed RHS
MAXbecause the signed edge list has 12 entries andMAXis at index 11.
Right. With 12 edges, row 12x + y is null exactly when y % 3 == 2. So the array-array and one-row checks never paired a valid left value with a right value of MIN / 2, 0, MAX / 2, or MAX. test_checked_integer_op now runs those checks with the mask and without it.
andygrove
reviewed
Nov 13, 2026
andygrove
left a comment
andygrove
left a comment
Member
There was a problem hiding this comment.
Thanks for this. Making the no-overflow path fast at the cost of a slower error path seems like the right trade-off to me. I checked the signed overflow detection, the widening multiply, slicing, scalar and null-scalar handling, and the length-mismatch fallback against the existing try_binary/try_unary behaviour, and they all look correct.
A few things to address:
1. Null slots now hold wrapped values instead of zero (numeric.rs:420)
try_unary/try_binary zero-fill null slots, while checked add/sub/mul now leave the wrapped result there. Nobody should be reading null slots, so this is fine, but please mention the behaviour change in the PR description.
2. Overflow in a null slot makes successful queries slower (numeric.rs:405)
Any overflow flag forces the fallback, including one from a null slot, even though the checked kernel would succeed. When inputs have extreme values in null slots, for example output from add_wrapping, every batch does the wrapping pass, allocates and drops the Vec, and reruns try_binary. That is slower than before this PR, as the null_slot_overflow benchmark shows. Could the overflow flag be masked with the validity bitmap before deciding to fall back? If that hurts vectorization, please at least list this case in the PR description as an input that gets slower.
3. Unsigned and Int64 mul (numeric.rs:376)
The comments say unsigned mul_overflowing stays scalar either way, and I'd expect the i64→i128 widening multiply not to vectorize either. If so, these types get the extra code size (97 KB vs 19 KB) and a second pass on overflow with little gain. Could you share benchmark numbers for them, or leave them on try_op!?
Smaller points / follow-ups
overflowing_op(numeric.rs:403) is a third copy of the scalar/array dispatch fromop!/try_op!, and it already differs fromtry_binary: it usesnulls()rather thanlogical_nulls(), and there's no empty-array early return. Can it reuse the existing dispatch, or at least match it?- The fast path only covers
integer_op(numeric.rs:579). Checked arithmetic on durations and timestamps, and directtry_binary(.., add_checked)callers, don't benefit. A generic helper inarity.rscould be a follow-up. - The speedup depends on
#[inline(never)]oncollect_overflowingkeeping the overflow flag in a register (numeric.rs:450). A short comment explaining that, and pointing to the benchmark that would catch a regression, would help future readers.
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (8aa2750) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^u?int
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-ez8gwx9u-593cc0f5-qcgr Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (8aa2750) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^u?int
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-efwz0fox-c189fe9c-8jv9 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (8aa2750) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^u?int
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1x7drz99-4b7c01e4-ppq8 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (8aa2750) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^u?int
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1x7drz99-4b7c01e4-sn47 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (8aa2750) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^u?int
RUSTFLAGS: -C target-cpu=x86-64-v3
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1x7drz99-ecac925a-k54z Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (8aa2750) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^u?int
RUSTFLAGS: -C target-cpu=x86-64-v3 -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1x7drz99-4b7c01e4-rh5x Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
|
Thanks @andygrove and @sunchao!
The description says it under "What changes are included in this PR?": "Null slots of the result hold the wrapping result instead of zero, as they already do for
Yes. After a pass that flagged an overflow,
Unsigned
On code size,
It matches it.
I filed #12241 for durations, which go through
The comment on |
Contributor
Author
|
run benchmark arithmetic_kernels |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark arithmetic_kernels |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C target-cpu=x86-64-v3
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
Contributor
Author
|
run benchmark arithmetic_kernels |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1jwtivje-1287609b-b8hd Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: arithmetic_kernels Sharding factor: 1 (1 workers). Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C target-cpu=x86-64-v3 -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1w1y66a2-83be8f18-tdfb Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1w1y66a2-83be8f18-trzm Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1jwtivje-04701ce0-x6d6 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C target-cpu=x86-64-v3
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1jwtivje-ad4e1533-4nvc Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: arithmetic_kernels Comparing vectorize-checked-arithmetic (6035cda) to 48ca288 diff Run configurationrun benchmark arithmetic_kernels
baseline:
ref: 48ca288b6b
env:
BENCH_FILTER: ^int64_(multiply|add)
RUSTFLAGS: -C target-cpu=x86-64-v3 -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
resources:
arch: amd64
shards: 1
DetailsPer-runner informationarithmetic_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1jwtivje-1287609b-b8hd Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench arithmetic_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
Author
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A.
Rationale for this change
Checked
add,subandmulon integers calltry_binaryortry_unary, which return at the first overflow, so LLVM doesn't vectorize them. DataFusion calls them whenfail_on_overflowis set (binary.rs), and Comet calls them for Spark's ANSI mode (checked_arithmetic.rs).What changes are included in this PR?
add,subandmulcompute the wrapping result and an overflow flag for every slot in one pass.0 - 1in a slot that a builder filled with 0, doesn't rerun the batch.addandsubtest sign bits, because the signedoverflowing_addandoverflowing_subdon't vectorize.i64mulusesoverflowing_mul, while the narrower signed types multiply in the next wider type. Widened toi128, thei64multiply stayed a scalarimulwhile LLVM packed each row's overflow flag into vector registers, and it measured 0.65x withx86-64-v3.Vec's spare capacity and callsset_lenwith the number of slots it wrote. Withcollect, LLVM sometimes left the loop insidecollectout of line, where it loaded and stored the overflow flag on every row. perf(arrow-arith): vectorize checked decimaladd,subandmul#12062 measured that at 0.83x for rescaledDecimal128add on Neoverse-V2. perf(arrow-arith): vectorize checked decimaladd,subandmul#12062 has the same helper.Vec::try_reserve_exact, so an allocation failure returnsArrowError::MemoryError.add_wrapping.Are these changes tested?
New tests in
arrow-arith/src/numeric.rscompare the kernels with the scalar checked operations at the edge values of every integer type. They run as arrays with and without nulls, as one-row slices, and with a scalar on either side. Another test moves one overflowing row through a 300-row batch, so that each lane of the vector loop and the scalar remainder hold it once.The overflow cases are expected to be slower, because they pay for the new pass, the null-slot check, and then the checked kernel. With
fail_on_overflowor ANSI mode, an overflow fails the query, so that cost is paid at most once per partition. A batch that overflows only in null slots pays for the pass and the null-slot check, and still beatsmainin the default-layout runs:int32_add_checked_null_slot_overflowmeasured 2.67x on Neoverse-V2, 1.36x on Xeon 8581C, and 2.76x withx86-64-v3. The lowest isuint64_add_checked_null_slot_overflowon Xeon 8581C, at 1.03x default and 0.86x aligned.The extra pass also costs code size: 135 KiB for
integer_opand the new helpers across the eight integer types, against 18.5 KiB before, in the thin-LTO bench binary on aarch64. The checked kernels that the fallback calls are unchanged.Bot runs: Neoverse-V2 default and aligned, Xeon 8581C default and aligned, and Xeon 8581C with
-C target-cpu=x86-64-v3default and aligned. Those runs measured 8aa2750. 6035cda changes only thei64multiply, and itsInt64runs are: Neoverse-V2 default and aligned, Xeon 8581C default and aligned, andx86-64-v3default and aligned.Are there any significant user-facing changes?
No.
AI Disclosure
I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.