Skip to content

Commit 306be7b

Browse files
committed
bench: address review - keep benchmark names, trim comments
- Keep the original benchmark names instead of adding a _tight_loop suffix. Renaming orphans the CodSpeed history for each id and adds to the skipped-benchmark backlog from #8807; the names still describe what is measured, and the batching is documented at the constant. - Trim the measurement write-ups to the durable reason. The instruction counts are specific to one machine and instrument, and already live in the previous commit message. - One line for the CUDA paths filter comment. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
1 parent 0f8eec7 commit 306be7b

3 files changed

Lines changed: 12 additions & 22 deletions

File tree

‎.github/workflows/codspeed.yml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,7 @@ jobs:
4242
filters: |
4343
cuda:
4444
- "vortex-cuda/**"
45-
# Only this workflow defines the CUDA benchmark jobs. Matching every workflow file
46-
# ran the walltime CUDA suite on unrelated CI edits (dependency bumps), where it
47-
# reported large changes that the hosted-runner walltime instrument cannot measure
48-
# reliably in the first place.
45+
# Only this workflow defines the CUDA benchmark jobs.
4946
- ".github/workflows/codspeed.yml"
5047
5148
bench-codspeed:

‎vortex-array/benches/search_sorted.rs‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,13 @@ fn main() {
1515
divan::main();
1616
}
1717

18-
/// One search over 65536 elements is ~16 comparisons, which measured 206 instructions per
19-
/// iteration against a fixed ~146 of `bench_refs` harness. At ~71% harness the two
20-
/// implementations below came out 203 vs 206, indistinguishable, and a small change in harness
21-
/// cost moved the reported figure by over 10%. Search a batch per iteration so the searches
22-
/// dominate; the targets differ, so the calls cannot be folded together.
18+
/// One search over 65536 elements is ~16 comparisons, small enough that the measurement is mostly
19+
/// harness overhead and the two implementations below become indistinguishable. Search a batch per
20+
/// iteration instead; the targets differ, so the calls cannot be folded together.
2321
const SEARCH_TARGETS: usize = 64;
2422

2523
#[divan::bench]
26-
fn binary_search_tight_loop_std(bencher: Bencher) {
24+
fn binary_search_std(bencher: Bencher) {
2725
let (sorted_array, targets) = fixture();
2826
bencher
2927
.with_inputs(|| (&sorted_array, &targets))
@@ -37,7 +35,7 @@ fn binary_search_tight_loop_std(bencher: Bencher) {
3735
}
3836

3937
#[divan::bench]
40-
fn binary_search_tight_loop_vortex(bencher: Bencher) {
38+
fn binary_search_vortex(bencher: Bencher) {
4139
let (sorted_array, targets) = fixture();
4240
bencher
4341
.with_inputs(|| (&sorted_array, &targets))

‎vortex-buffer/benches/vortex_bitbuffer.rs‎

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -153,15 +153,13 @@ fn value_arrow_buffer(bencher: Bencher, length: usize) {
153153
});
154154
}
155155

156-
/// Slicing only adjusts an offset, a length and a refcount, so its cost is independent of the
157-
/// buffer length: every entry in [`INPUT_SIZE`] measured exactly 125 instructions per iteration,
158-
/// almost all of it harness. Measure one representative length, and enough slices per iteration
159-
/// that the slicing dominates.
156+
/// Slicing only adjusts an offset, a length and a refcount, so its cost is independent of buffer
157+
/// length. Measure one length, with enough slices per iteration to stay above harness overhead.
160158
const SLICE_ITERS: usize = 64;
161159
const SLICE_INPUT_SIZE: usize = 65_536;
162160

163161
#[divan::bench]
164-
fn slice_tight_loop_vortex_buffer(bencher: Bencher) {
162+
fn slice_vortex_buffer(bencher: Bencher) {
165163
let buffer = BitBuffer::from_iter((0..SLICE_INPUT_SIZE).map(|i| i % 2 == 0));
166164
let mid = SLICE_INPUT_SIZE / 2;
167165
bencher.with_inputs(|| &buffer).bench_refs(|buffer| {
@@ -175,7 +173,7 @@ fn slice_tight_loop_vortex_buffer(bencher: Bencher) {
175173

176174
#[cfg(not(codspeed))]
177175
#[divan::bench]
178-
fn slice_tight_loop_arrow_buffer(bencher: Bencher) {
176+
fn slice_arrow_buffer(bencher: Bencher) {
179177
let buffer = Arrow(BooleanBuffer::from_iter(
180178
(0..SLICE_INPUT_SIZE).map(|i| i % 2 == 0),
181179
));
@@ -187,11 +185,8 @@ fn slice_tight_loop_arrow_buffer(bencher: Bencher) {
187185
});
188186
}
189187

190-
/// A popcount costs ~0.04 instructions per bit, while one `bench_refs` iteration carries a fixed
191-
/// ~146 instructions of harness. Measured per-iteration instruction counts were 187 at 1024 bits
192-
/// and 228 at 2048, i.e. 78% and 64% harness, so a small change in harness cost moved the
193-
/// reported figure by more than 10% while the popcount was unchanged. At 16384 and 65536 bits
194-
/// (802 and 2770 instructions) the harness is 18% and 5%, which leaves the count dominant.
188+
/// Below a few thousand bits the measurement is mostly divan's fixed per-iteration overhead
189+
/// rather than the popcount, so it moves when the count itself has not changed.
195190
const TRUE_COUNT_INPUT_SIZE: &[usize] = &[16_384, 65_536];
196191

197192
#[divan::bench(args = TRUE_COUNT_INPUT_SIZE)]

0 commit comments

Comments
 (0)