Skip to content

Commit eed0212

Browse files
Measure a subset of four more benchmark sweeps on CodSpeed (#11132)
## Summary Follow-up to #11131, using the same pattern. Four more parameter sweeps keep every case for local runs and measure a representative subset under `cfg(codspeed)`. CodSpeed cases that are kept keep their names, so their history continues. The cases to keep were chosen from one local walltime run of every case. Dimensions whose timings scale linearly or barely move were reduced; dimensions where timings jump (algorithm or threshold changes) were kept. ## Changes | Bench | CodSpeed benchmarks, before → after | What CodSpeed leaves out | |---|---:|---| | `vortex-fastlanes/canonicalize_bench` | 32 → 14 | The 100-chunk cases, which cost about 30× the 10-chunk ones; `(1000, 1000, 0.00)` already covers the same total length. Also the 1% patch fraction, which lies between 0% and 10%. | | `vortex-array/dict_compare` | 27 → 16 | The 64-value dictionary in the plain compares, which time alike for every dictionary size. The sliced sweep keeps its ends, its middle, and both sides of codes as long as values (`9_999`, `10_000`). | | `vortex-fastlanes/compute_between` | 33 → 11 | Two of the three lengths, which scale linearly. Every type is still measured. | | `vortex-array/filter_bool` | 61 → 49 | The 250,000-row size. The density sweeps, which change algorithm with density, are kept in full. | The estimated saving is about 6 s of CodSpeed simulation time, mostly in shard 7 (FastLanes & decimals), from `canonicalize_bench`'s 100-chunk cases. Left alone on purpose: - `fsst_take`: its sweep sits on either side of the compressed-bytes threshold, and timings jump around rather than scaling. - `take_chunked`: its cases are hand-tuned. - `take_filter`, `widen`: each case measures a distinct code path. Verification: built every changed bench with and without `--cfg codspeed` and ran each binary to count the benchmarks it runs. The local builds run 32 / 27 / 33 / 61 and the CodSpeed builds 14 / 16 / 11 / 49, with no compiler warnings. I didn't run clippy, rustfmt or tests. --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniBAqXtmDDNUP9pHsfSo)_ Signed-off-by: Joe Isaacs <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 2d128af commit eed0212

9 files changed

Lines changed: 97 additions & 62 deletions

File tree

‎encodings/alp/benches/alp_compress.rs‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,6 @@ fn main() {
3838
divan::main();
3939
}
4040

41-
// CodSpeed measures a subset. Times scale linearly with length, and the two nullable validity
42-
// fractions time alike, so CodSpeed keeps one small length and one nullable fraction.
4341
const BENCH_ARGS: &[(usize, f64, f64)] = if cfg!(codspeed) {
4442
&[
4543
// length, fraction_patch, fraction_valid

‎encodings/fastlanes/benches/bitpack_compare_sweep.rs‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@
33

44
//! Sweeps the public `BitPackedArray` compare-against-constant path (`array.binary(rhs, op)`) over
55
//! every integer type and every valid bit width, so a kernel change shows up as a CodSpeed diff.
6-
//! CodSpeed runs a sample of the widths (see [`CODSPEED_WIDTHS`]); local runs cover all of them.
76
//!
87
//! The array holds in-range values (no patches, no out-of-range fast path), so each iteration runs
98
//! the full unpack + per-element compare kernel that backs `<BitPacked as CompareKernel>`.
@@ -61,10 +60,6 @@ static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
6160
/// Number of elements per benchmarked array
6261
const LEN: usize = if cfg!(codspeed) { 8 * 1024 } else { 64 * 1024 };
6362

64-
/// Bit widths that CodSpeed measures, each type taking those below its native width. They sit
65-
/// just off the powers of two, plus width 1, so that each type includes its widest valid width.
66-
/// Signed and unsigned types of one width share their fused kernels, and the FastLanes unpack
67-
/// code for every width comes from one generator, so this sample stands in for the full sweep.
6863
const CODSPEED_WIDTHS: &[usize] = &[1, 3, 5, 7, 9, 15, 17, 31, 33, 63];
6964

7065
/// The bit widths to benchmark for a type of `native_bits` bits. Bit-packing requires the target

‎encodings/fastlanes/benches/canonicalize_bench.rs‎

Lines changed: 41 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -29,19 +29,29 @@ fn main() {
2929
divan::main();
3030
}
3131

32-
const BENCH_ARGS: &[(usize, usize, f64)] = &[
33-
// chunk_len, chunk_count, fraction_patched
34-
(10000, 1, 0.10),
35-
(10000, 1, 0.01),
36-
(10000, 1, 0.00),
37-
(10000, 10, 0.10),
38-
(10000, 10, 0.01),
39-
(10000, 10, 0.00),
40-
(10000, 100, 0.10),
41-
(10000, 100, 0.01),
42-
(10000, 100, 0.00),
43-
(1000, 1000, 0.00),
44-
];
32+
const BENCH_ARGS: &[(usize, usize, f64)] = if cfg!(codspeed) {
33+
&[
34+
// chunk_len, chunk_count, fraction_patched
35+
(10000, 1, 0.10),
36+
(10000, 10, 0.10),
37+
(10000, 10, 0.00),
38+
(1000, 1000, 0.00),
39+
]
40+
} else {
41+
&[
42+
// chunk_len, chunk_count, fraction_patched
43+
(10000, 1, 0.10),
44+
(10000, 1, 0.01),
45+
(10000, 1, 0.00),
46+
(10000, 10, 0.10),
47+
(10000, 10, 0.01),
48+
(10000, 10, 0.00),
49+
(10000, 100, 0.10),
50+
(10000, 100, 0.01),
51+
(10000, 100, 0.00),
52+
(1000, 1000, 0.00),
53+
]
54+
};
4555

4656
static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
4757
let session = VortexSession::empty()
@@ -122,15 +132,24 @@ fn canonical_into_non_nullable(
122132
});
123133
}
124134

125-
const NULLABLE_BENCH_ARGS: &[(usize, usize, f64)] = &[
126-
// chunk_len, chunk_count, fraction_patched
127-
(10000, 1, 0.10),
128-
(10000, 1, 0.00),
129-
(10000, 10, 0.10),
130-
(10000, 10, 0.00),
131-
(10000, 100, 0.10),
132-
(10000, 100, 0.00),
133-
];
135+
const NULLABLE_BENCH_ARGS: &[(usize, usize, f64)] = if cfg!(codspeed) {
136+
&[
137+
// chunk_len, chunk_count, fraction_patched
138+
(10000, 1, 0.10),
139+
(10000, 10, 0.10),
140+
(10000, 10, 0.00),
141+
]
142+
} else {
143+
&[
144+
// chunk_len, chunk_count, fraction_patched
145+
(10000, 1, 0.10),
146+
(10000, 1, 0.00),
147+
(10000, 10, 0.10),
148+
(10000, 10, 0.00),
149+
(10000, 100, 0.10),
150+
(10000, 100, 0.00),
151+
]
152+
};
134153

135154
#[divan::bench(args = NULLABLE_BENCH_ARGS)]
136155
fn into_canonical_nullable(

‎encodings/fastlanes/benches/compute_between.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,11 @@ fn generate_alp_bit_pack_primitive_array<T: NativePType + NumCast>(
8888
ALP::new(bp, alp.exponents(), None).into_array()
8989
}
9090

91-
const BENCH_ARGS: &[usize] = &[2 << 10, 2 << 13, 2 << 14];
91+
const BENCH_ARGS: &[usize] = if cfg!(codspeed) {
92+
&[2 << 13]
93+
} else {
94+
&[2 << 10, 2 << 13, 2 << 14]
95+
};
9296

9397
mod primitive {
9498
use divan::Bencher;

‎encodings/fsst/benches/fsst_compress.rs‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,6 @@ static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
5151
});
5252

5353
// [(string_count, avg_len, unique_chars)]
54-
//
55-
// CodSpeed measures a subset. Times scale linearly with `string_count`, and `unique_chars` moves
56-
// them by less than 20%, so CodSpeed keeps one string count and checks the alphabet size once.
5754
const BENCH_ARGS: &[(usize, usize, u8)] = if cfg!(codspeed) {
5855
&[(500, 4, 4), (500, 16, 4), (500, 64, 4), (500, 64, 8)]
5956
} else {
@@ -149,8 +146,6 @@ fn canonicalize_compare(
149146
}
150147

151148
// [(chunk_size, string_count, avg_len, unique_chars)]
152-
//
153-
// CodSpeed measures one case per chunk shape: `unique_chars` moves these times by about 10%.
154149
const CHUNKED_BENCH_ARGS: &[(usize, usize, usize, u8)] = if cfg!(codspeed) {
155150
&[(20, 100, 16, 4), (20, 50, 8, 4), (10, 150, 64, 4)]
156151
} else {
@@ -199,8 +194,6 @@ fn chunked_into_canonical(
199194
.bench_refs(|(array, ctx)| array.clone().execute::<Canonical>(ctx));
200195
}
201196

202-
/// A value that occurs in `array`, so the compare matches some rows and the needle compresses to
203-
/// symbols from the trained table rather than to escapes.
204197
fn needle(array: &ArrayRef, ctx: &mut ExecutionCtx) -> Scalar {
205198
array.execute_scalar(0, ctx).unwrap()
206199
}

‎vortex-array/benches/dict_compare.rs‎

Lines changed: 35 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -42,12 +42,20 @@ static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
4242
.with::<MemorySession>()
4343
});
4444

45-
const LENGTH_AND_UNIQUE_VALUES: &[(usize, usize)] = &[
46-
// length, unique_values
47-
(50_000, 2),
48-
(50_000, 64),
49-
(50_000, 2048),
50-
];
45+
const LENGTH_AND_UNIQUE_VALUES: &[(usize, usize)] = if cfg!(codspeed) {
46+
&[
47+
// length, unique_values
48+
(50_000, 2),
49+
(50_000, 2048),
50+
]
51+
} else {
52+
&[
53+
// length, unique_values
54+
(50_000, 2),
55+
(50_000, 64),
56+
(50_000, 2048),
57+
]
58+
};
5159

5260
#[divan::bench(args = LENGTH_AND_UNIQUE_VALUES)]
5361
fn bench_compare_primitive(bencher: divan::Bencher, (len, uniqueness): (usize, usize)) {
@@ -108,17 +116,27 @@ fn bench_compare_varbinview(bencher: divan::Bencher, (len, uniqueness): (usize,
108116
})
109117
}
110118

111-
const CODES_AND_VALUES_LENGTHS: &[(usize, usize)] = &[
112-
(1_000, 10_000),
113-
(2_000, 10_000),
114-
(2_500, 10_000),
115-
(3_333, 10_000),
116-
(5_000, 10_000),
117-
(7_500, 10_000),
118-
(9_999, 10_000),
119-
(10_000, 10_000),
120-
(20_000, 10_000),
121-
];
119+
const CODES_AND_VALUES_LENGTHS: &[(usize, usize)] = if cfg!(codspeed) {
120+
&[
121+
(1_000, 10_000),
122+
(5_000, 10_000),
123+
(9_999, 10_000),
124+
(10_000, 10_000),
125+
(20_000, 10_000),
126+
]
127+
} else {
128+
&[
129+
(1_000, 10_000),
130+
(2_000, 10_000),
131+
(2_500, 10_000),
132+
(3_333, 10_000),
133+
(5_000, 10_000),
134+
(7_500, 10_000),
135+
(9_999, 10_000),
136+
(10_000, 10_000),
137+
(20_000, 10_000),
138+
]
139+
};
122140

123141
#[divan::bench(args = CODES_AND_VALUES_LENGTHS)]
124142
fn bench_compare_sliced_dict_primitive(

‎vortex-array/benches/dict_compress.rs‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,7 @@ fn main() {
3131
divan::main();
3232
}
3333

34-
// CodSpeed measures a subset. Times scale linearly with length, and 2, 4 and 8 unique values time
35-
// alike, so CodSpeed keeps one length and one small dictionary. Lengths are sized to keep CodSpeed
36-
// simulation under 1ms per benchmark.
34+
// Lengths sized to keep CodSpeed simulation under 1ms per benchmark.
3735
const BENCH_ARGS: &[(usize, usize)] = if cfg!(codspeed) {
3836
&[
3937
// length, unique_values

‎vortex-array/benches/filter_bool.rs‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,11 @@ static SESSION: LazyLock<VortexSession> =
4141
/// 1000 rows took less than 1 µs, so the allocator work left behind by the previous, larger case
4242
/// was a large part of the single measured iteration in simulation, and CodSpeed reported no
4343
/// execution time for `filter_powerlaw_by_mostly_false[1000]`.
44-
const SIZES: &[usize] = &[10_000, 100_000, 250_000];
44+
const SIZES: &[usize] = if cfg!(codspeed) {
45+
&[10_000, 100_000]
46+
} else {
47+
&[10_000, 100_000, 250_000]
48+
};
4549
const DENSITY_SWEEP_SIZE: usize = 100_000;
4650
const ARRAY_SEED: u64 = 42;
4751
const MASK_SEED: u64 = 43;
@@ -421,9 +425,17 @@ fn density_sweep_single_slice(bencher: Bencher) {
421425

422426
// --- Extreme cases ---
423427

424-
const LARGE_SIZES: &[usize] = &[10_000, 100_000, 250_000];
428+
const LARGE_SIZES: &[usize] = if cfg!(codspeed) {
429+
&[10_000, 100_000]
430+
} else {
431+
&[10_000, 100_000, 250_000]
432+
};
425433
const ALL_TRUE_SIZES: &[usize] = &[100_000];
426-
const ULTRA_SPARSE_SIZES: &[usize] = &[100_000, 250_000];
434+
const ULTRA_SPARSE_SIZES: &[usize] = if cfg!(codspeed) {
435+
&[100_000]
436+
} else {
437+
&[100_000, 250_000]
438+
};
427439

428440
#[divan::bench(args = ALL_TRUE_SIZES)]
429441
fn filter_all_true(bencher: Bencher, n: usize) {

‎vortex-array/benches/run_end_compress.rs‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,6 @@ static SESSION: LazyLock<VortexSession> = LazyLock::new(|| {
4646
// (length, run_step). The (4_000, 1024) point is left out: it flipped by 20% between runs of
4747
// identical code on more than 40 pull requests, and (10_000, 1024) and (10_000, 4096) keep long
4848
// runs covered.
49-
//
50-
// CodSpeed measures every run step at the longest length only: times scale with length.
5149
const BENCH_ARGS: &[(usize, usize)] = if cfg!(codspeed) {
5250
&[
5351
(10_000, 4),

0 commit comments

Comments
 (0)