Skip to content

Commit e482701

Browse files
connortsui20claude
andauthored
bench: keep third-party and frozen baselines out of CodSpeed (#9952)
## Summary Second of three stacked PRs on CodSpeed flakiness. The first resizes benchmarks, the third unifies the allocator. The three noisiest series in the whole suite are benchmarks of code Vortex does not own or does not change: | Benchmark | Flips on unrelated PRs (all PRs) | Max swing | |---|---:|---:| | `lane_kernels.rs`: `arrow_checked_add_u32` (neon leg) | 64 (97) | 67% | | `lane_kernels.rs`: `arrow_checked_add_u32` (avx2 leg) | 48 (65) | 22% | | `collect_bool.rs`: `words_gather_scalar[65536]` (avx2 leg) | 45 (62) | 15% | | `lane_kernels.rs`: `arrow_checked_add_u32` (avx512 leg) | 30 (41) | 22% | `arrow_checked_add_u32` runs the arrow-rs checked add over the same data as `lanezip_checked_add_u32`. `words_gather_scalar` runs a frozen copy of the previous scalar `collect_bool` loop. Neither can regress because of a pull request, so a change in their number is never actionable, and on the walltime legs they flipped on about half of all pull requests. ## Changes Both benchmarks stay in the tree for local `cargo bench` comparisons, but leave CodSpeed: they lose `#[cpu_features]`, which takes them off the walltime legs, and gain `#[cfg(not(codspeed))]`, which keeps them out of the simulation build. Imports and fixture fields that only they use are gated the same way. The guide gets a short section on keeping third-party and frozen baselines out of CodSpeed. The other arrow baselines (`arrow_narrow_*` and `arrow_widen_*` in `lane_kernels.rs`) run in simulation, have not flipped, and are left as they are. Checks run: - `cargo clippy --locked --benches -p vortex-buffer -p vortex-compute -- -D warnings` - `RUSTFLAGS="--cfg codspeed -D warnings" cargo check --locked --benches -p vortex-buffer -p vortex-compute`, to confirm the gated build has no unused items - `cargo +nightly-2026-09-10 fmt` on the touched crates 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn --- _Generated by [Claude Code](https://claude.ai/code)_ Signed-off-by: Claude <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent 5c7f370 commit e482701

3 files changed

Lines changed: 33 additions & 11 deletions

File tree

‎docs/developer-guide/benchmarking.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,15 @@ benchmark is about kernel code rather than memory bandwidth.
159159

160160
Use `#[cfg(not(codspeed))]` for benchmarks that are incompatible with CodSpeed.
161161

162+
### Keep third-party and frozen baselines out of CodSpeed
163+
164+
A benchmark of code that Vortex does not own, such as an arrow-rs kernel over the same data, or
165+
of a frozen copy of an old Vortex implementation, cannot regress because of a pull request, so a
166+
change in its number is never actionable. On the walltime legs these baselines were among the
167+
noisiest series in the suite. Keep them for local `cargo bench` comparisons, but gate them with
168+
`#[cfg(not(codspeed))]` and leave them untagged, as `vortex-compute/benches/lane_kernels.rs` and
169+
`vortex-buffer/benches/collect_bool.rs` do.
170+
162171
### CodSpeed's single-run model
163172

164173
CI benchmarks run under [CodSpeed's CPU simulation](https://codspeed.io/docs/instruments/cpu),

‎vortex-buffer/benches/collect_bool.rs‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,13 @@
1313
//! - `collect_bool_*` / `from_bool_slice`: the public entry points end to end, with a
1414
//! boolean-gather predicate and a `u32` comparison predicate.
1515
//!
16-
//! `words_gather_dispatch` and `words_gather_scalar` carry `#[cpu_features]`, so they are
17-
//! measured on every walltime CPU-feature leg rather than in simulation. Both are written
18-
//! once and compiled differently per leg: the shipped entry point picks its pack kernel
19-
//! through `cfg(target_feature)`, and how well the scalar loop auto-vectorizes depends on
20-
//! the build. Comparing them across legs is the point.
16+
//! `words_gather_dispatch` carries `#[cpu_features]`, so it is measured on every walltime
17+
//! CPU-feature leg rather than in simulation: the shipped entry point is written once and
18+
//! picks its pack kernel through `cfg(target_feature)`, so each leg measures a different
19+
//! build of it. `words_gather_scalar`, the frozen copy of the previous scalar loop, is
20+
//! compiled only outside CodSpeed: its code never changes, so a change in its walltime is
21+
//! never actionable, and it flipped by up to 15% between runs of identical code. Run
22+
//! `cargo bench` locally to compare the two under one set of build flags.
2123
//!
2224
//! The hand-written per-kernel benchmarks are not tagged. Each one needs an instruction set
2325
//! extension the other legs do not build for, so they stay out of CodSpeed entirely and
@@ -27,6 +29,7 @@
2729
2830
use divan::Bencher;
2931
use vortex_buffer::BitBuffer;
32+
#[cfg(not(codspeed))]
3033
use vortex_buffer::collect_bool_word_scalar;
3134
#[cfg(not(codspeed))]
3235
use vortex_buffer::pack_bool_word_swar;
@@ -123,7 +126,7 @@ fn words_gather_dispatch(bencher: Bencher, len: usize) {
123126
});
124127
}
125128

126-
#[vortex_bench_support::cpu_features]
129+
#[cfg(not(codspeed))]
127130
#[divan::bench(args = GATHER_INPUT_SIZE)]
128131
fn words_gather_scalar(bencher: Bencher, len: usize) {
129132
bench_words_gather(bencher, len, |words, len, bools| {
@@ -180,6 +183,7 @@ fn words_gather_neon(bencher: Bencher, len: usize) {
180183

181184
/// Faithful copy of the previous scalar-only `collect_bool_words` word loop, used as the
182185
/// baseline for the end-to-end comparison.
186+
#[cfg(not(codspeed))]
183187
fn collect_bool_words_old(words: &mut [u64], len: usize, mut f: impl FnMut(usize) -> bool) {
184188
let full = len / 64;
185189
let remainder = len % 64;

‎vortex-compute/benches/lane_kernels.rs‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,21 +13,26 @@
1313
//! equivalent arrow-rs kernel over the same data shape, so the divan report
1414
//! lines up side-by-side.
1515
//!
16-
//! The checked-add pair carries `#[cpu_features]`, so both are measured on every
17-
//! walltime CPU-feature leg rather than in simulation: they are one lane loop
18-
//! compiled differently per leg, and comparing them against arrow-rs is only
19-
//! meaningful under the same build flags. The cast benches stay in simulation.
16+
//! `lanezip_checked_add_u32` carries `#[cpu_features]`, so it is measured on every
17+
//! walltime CPU-feature leg rather than in simulation: it is one lane loop compiled
18+
//! differently per leg. Its arrow-rs sibling is compiled only outside CodSpeed. The
19+
//! arrow kernel is not Vortex code, so a change in its walltime is never actionable,
20+
//! and it flipped by up to 67% between runs of identical code. Run `cargo bench`
21+
//! locally to compare the pair under the same build flags. The cast benches stay in
22+
//! simulation.
2023
2124
#![expect(clippy::unwrap_used)]
2225
#![expect(clippy::clone_on_ref_ptr)]
2326

2427
use std::mem::MaybeUninit;
2528
use std::sync::Arc;
2629

30+
#[cfg(not(codspeed))]
2731
use arrow_arith::numeric::add;
2832
use arrow_array::ArrayRef as ArrowArrayRef;
2933
use arrow_array::Int32Array;
3034
use arrow_array::UInt16Array;
35+
#[cfg(not(codspeed))]
3136
use arrow_array::UInt32Array;
3237
use arrow_array::UInt64Array;
3338
use arrow_buffer::NullBuffer;
@@ -289,7 +294,9 @@ struct AddFixture {
289294
rhs_mask: BitBuffer,
290295
/// Plain `Vec<bool>` mirrors of the validity masks — used to build the arrow
291296
/// `NullBuffer`s for the baseline bench.
297+
#[cfg(not(codspeed))]
292298
lhs_valid: Vec<bool>,
299+
#[cfg(not(codspeed))]
293300
rhs_valid: Vec<bool>,
294301
}
295302

@@ -333,7 +340,9 @@ fn add_fixture(n: usize) -> AddFixture {
333340
rhs,
334341
lhs_mask,
335342
rhs_mask,
343+
#[cfg(not(codspeed))]
336344
lhs_valid,
345+
#[cfg(not(codspeed))]
337346
rhs_valid,
338347
}
339348
}
@@ -361,7 +370,7 @@ fn lanezip_checked_add_u32(bencher: Bencher, n: usize) {
361370
});
362371
}
363372

364-
#[vortex_bench_support::cpu_features]
373+
#[cfg(not(codspeed))]
365374
#[divan::bench(args = ADD_SIZES)]
366375
fn arrow_checked_add_u32(bencher: Bencher, n: usize) {
367376
let f = add_fixture(n);

0 commit comments

Comments
 (0)