Skip to content

Commit 0fea4a3

Browse files
connortsui20claude
andauthored
bench: stop gating on benchmarks CodSpeed cannot measure (#10063)
## Summary Four groups of benchmarks that cannot tell a reviewer anything about a pull request. Every value below moved on pull requests that changed no Vortex code. | Benchmark | Values seen | Unrelated PRs | | --- | --- | ---: | | `allocation::allocate_drop_arrow[65536]` | 90.7 / 101.8 µs | 4 | | `allocation::from_vec_drop_arrow[16384]` | 40.4 / 31.4 µs | 4 | | `take_fsl::..._force_manual_range_copy[2048, 10]` | 10.2 / 63.3 µs | 7 | | `take_fsl::..._force_per_index[256, 10]` | 40 / 56.9 µs | 4 | | `take_chunked::take[duplicates/repeated/...]` | 24 / 104 µs | 3 | | `take_slices_to_buffer_matrix` (3 cases) | no execution time | 26 of 26 | | `row_fn_utf8::decode` (both legs) | 80.7 / 187.6 µs | 7 | | `row_fn_output::filtered_*` (all legs) | 10 to 19% swings | 5 | ## Changes - **allocation**: gate the `bytes` and Arrow cases out of CodSpeed. They measure allocators Vortex does not own, so no pull request here can move them. They stay for local `cargo bench`. - **take_fsl**: remove the three forced-strategy benchmarks. They compare internal strategies against each other, which is a local question, and at 10 indices the measured region is dominated by allocation. Two of the three flip, and both also report no execution time on some runs. - **take_chunked**: drop the `repeated` duplicates case, which took one index 1,000 times. `duplicate90` and `duplicate99` cover heavy duplication without the degenerate input. - **row_fn_utf8**: remove the file. Its one benchmark swung 10 to 18% per report on 13 of the last 24, including on a lockfile refresh and a one-file Python fix. Both legs move, so this is the benchmark rather than one runner. - **row_fn_output**: remove `filtered_owned_i64` and `filtered_sink_i64` and their support types. They swung 10 to 19% on avx2, avx512 and neon alike, on unrelated pull requests. The infallible and deferred benchmarks in the same file have stayed put and are untouched. - **take_slices_to_buffer_matrix**: read one output element through an opaque index. Black-boxing the returned buffer alone leaves the copy loop dead, because nothing reads what it wrote into spare capacity, which is why three cases have measured nothing on every recent report. ### Expect a one-time rebaseline Two kinds. The advancing-pointer numbers rise once their stores survive. The other kind is worth understanding: removing a benchmark speeds up the ones that follow it in the same binary. `allocate_drop_vortex` at 64, 256 and 1024 bytes improved by 20 to 28% on this branch, with no change to the code they measure. The run logs list benchmarks in execution order: those three run at positions 40 to 42 on develop and 30 to 32 here, and their immediate predecessor is the same on both. What is gone is ten earlier benchmarks, each doing 256 allocate-and-free cycles across five sizes. Two `take_chunked` cases moved the same way. The deltas are flat rather than proportional: 14.6, 14.4 and 18.7 µs in allocation, 39.1 and 40.1 µs in take_chunked. That is the fixed allocator cleanup cost from #9994 again. Only the small sizes moved, while 16 KiB and 64 KiB did not, which fits mimalloc's per-size-class bookkeeping: these benchmarks were partly measuring the mess their predecessors left behind. Checked with clippy under `-D warnings`, the same under `--cfg codspeed`, and `cargo fmt`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn --- _Generated by [Claude Code](https://claude.ai/code)_ --------- Signed-off-by: Claude <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent 21b3dea commit 0fea4a3

7 files changed

Lines changed: 24 additions & 440 deletions

File tree

‎vortex-array/Cargo.toml‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -149,10 +149,6 @@ harness = false
149149
name = "row_fn_bool_retry"
150150
harness = false
151151

152-
[[bench]]
153-
name = "row_fn_utf8"
154-
harness = false
155-
156152
[[bench]]
157153
name = "row_fn_output"
158154
harness = false

‎vortex-array/benches/row_fn_output.rs‎

Lines changed: 0 additions & 153 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@ use divan::Bencher;
1313
use divan::counter::ItemsCount;
1414
use mimalloc::MiMalloc;
1515
use vortex_array::ArrayRef;
16-
use vortex_array::ExecutionCtx;
1716
use vortex_array::IntoArray;
1817
use vortex_array::VortexSessionExecute;
1918
use vortex_array::array_session;
@@ -24,14 +23,9 @@ use vortex_array::dtype::NativePType;
2423
use vortex_array::scalar_fn::EmptyOptions;
2524
use vortex_array::scalar_fn::ScalarFnId;
2625
use vortex_array::scalar_fn::VecExecutionArgs;
27-
use vortex_array::scalar_fn::unstable::row::InitializedElement;
28-
use vortex_array::scalar_fn::unstable::row::InputElement;
2926
use vortex_array::scalar_fn::unstable::row::RowFn;
3027
use vortex_array::scalar_fn::unstable::row::RowVisitor;
31-
use vortex_array::scalar_fn::unstable::row::UninitElementSink;
3228
use vortex_array::scalar_fn::unstable::row::execute_rows;
33-
use vortex_array::validity::Validity;
34-
use vortex_buffer::Buffer;
3529
use vortex_error::VortexExpect;
3630
use vortex_error::VortexResult;
3731
use vortex_error::vortex_ensure;
@@ -62,36 +56,13 @@ const INPUT_SHAPES: &[InputShape] = &[InputShape::PerRowPerRow];
6256
/// and that is the difference worth watching.
6357
const CONSTANT_SHAPES: &[InputShape] = &[InputShape::PerRowConstant, InputShape::ConstantPerRow];
6458

65-
const VALIDITY_PATTERNS: &[ValidityPattern] = &[
66-
ValidityPattern::AllValid,
67-
ValidityPattern::OneNullInEight,
68-
ValidityPattern::NineNullsInTen,
69-
];
70-
7159
#[derive(Clone, Copy, Debug)]
7260
enum InputShape {
7361
PerRowPerRow,
7462
PerRowConstant,
7563
ConstantPerRow,
7664
}
7765

78-
#[derive(Clone, Copy, Debug)]
79-
enum ValidityPattern {
80-
AllValid,
81-
OneNullInEight,
82-
NineNullsInTen,
83-
}
84-
85-
impl ValidityPattern {
86-
fn is_valid(self, index: usize) -> bool {
87-
match self {
88-
Self::AllValid => true,
89-
Self::OneNullInEight => !index.is_multiple_of(8),
90-
Self::NineNullsInTen => index.is_multiple_of(10),
91-
}
92-
}
93-
}
94-
9566
trait BenchPrimitive: NativePType {
9667
fn per_row(offset: usize) -> ArrayRef;
9768

@@ -120,57 +91,6 @@ impl BenchPrimitive for i64 {
12091
}
12192
}
12293

123-
/// Primitive input that forces partially valid batches through the filtered fallback.
124-
///
125-
/// `DENSE_SAFE = false` selects valid-only execution. This type inherits the conservative
126-
/// null-tolerant decode implementation, so direct valid-row execution declines the original
127-
/// nullable input. A partially valid input must therefore use the executor's filtered fallback.
128-
struct FilteredI64;
129-
130-
// SAFETY: the view is a native slice, and its reported length is the slice length.
131-
unsafe impl InputElement for FilteredI64 {
132-
type Column = Buffer<i64>;
133-
type Constant = i64;
134-
type View<'a> = &'a [i64];
135-
type Elem<'a> = i64;
136-
137-
const DENSE_SAFE: bool = false;
138-
const DECODE_INFALLIBLE: bool = true;
139-
140-
fn validate(dtype: &DType) -> VortexResult<()> {
141-
<i64 as InputElement>::validate(dtype)
142-
}
143-
144-
fn decode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult<Self::Column> {
145-
<i64 as InputElement>::decode(array, ctx)
146-
}
147-
148-
fn decode_constant(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult<Self::Constant> {
149-
<i64 as InputElement>::decode_constant(array, ctx)
150-
}
151-
152-
fn get(column: &Self::Column, index: usize) -> Self::Elem<'_> {
153-
column[index]
154-
}
155-
156-
fn get_constant(constant: &Self::Constant) -> Self::Elem<'_> {
157-
*constant
158-
}
159-
160-
fn view(column: &Self::Column) -> Self::View<'_> {
161-
column.as_slice()
162-
}
163-
164-
fn get_from_view<'a>(view: &Self::View<'a>, index: usize) -> Self::Elem<'a> {
165-
view[index]
166-
}
167-
168-
unsafe fn get_from_view_unchecked<'a>(view: &Self::View<'a>, index: usize) -> Self::Elem<'a> {
169-
// SAFETY: forwarded from this method's contract.
170-
unsafe { *view.get_unchecked(index) }
171-
}
172-
}
173-
17494
#[derive(Clone)]
17595
struct InfallibleBool<T>(PhantomData<T>);
17696

@@ -180,12 +100,6 @@ struct DeferredBool<T>(PhantomData<T>);
180100
#[derive(Clone)]
181101
struct DeferredI64;
182102

183-
#[derive(Clone)]
184-
struct FilteredOwnedI64;
185-
186-
#[derive(Clone)]
187-
struct FilteredSinkI64;
188-
189103
impl<T: BenchPrimitive> RowFn for InfallibleBool<T> {
190104
type Options = EmptyOptions;
191105

@@ -267,51 +181,6 @@ impl RowFn for DeferredI64 {
267181
}
268182
}
269183

270-
impl RowFn for FilteredOwnedI64 {
271-
type Options = EmptyOptions;
272-
273-
const ARG_NAMES: &'static [&'static str] = &["value"];
274-
const INFALLIBLE: bool = true;
275-
276-
fn id(&self) -> ScalarFnId {
277-
static ID: CachedId = CachedId::new("bench.row_fn_output.filtered_owned_i64");
278-
*ID
279-
}
280-
281-
fn dispatch<V: RowVisitor>(
282-
&self,
283-
_options: &Self::Options,
284-
_args: &[DType],
285-
visitor: V,
286-
) -> VortexResult<V::VisitResult> {
287-
visitor.visit::<(FilteredI64,), i64>(|(value,)| value.wrapping_add(1))
288-
}
289-
}
290-
291-
impl RowFn for FilteredSinkI64 {
292-
type Options = EmptyOptions;
293-
294-
const ARG_NAMES: &'static [&'static str] = &["value"];
295-
const INFALLIBLE: bool = true;
296-
297-
fn id(&self) -> ScalarFnId {
298-
static ID: CachedId = CachedId::new("bench.row_fn_output.filtered_sink_i64");
299-
*ID
300-
}
301-
302-
fn dispatch<V: RowVisitor>(
303-
&self,
304-
_options: &Self::Options,
305-
_args: &[DType],
306-
visitor: V,
307-
) -> VortexResult<V::VisitResult> {
308-
visitor.visit_into::<(FilteredI64,), UninitElementSink<i64>, _>((), |(value,), output| {
309-
// SAFETY: `output` is the `UninitElementSink` row supplied for this callback.
310-
unsafe { InitializedElement::write(output, value.wrapping_add(1)) }
311-
})
312-
}
313-
}
314-
315184
#[vortex_bench_support::cpu_features]
316185
#[divan::bench(types = [i32, i64], args = INPUT_SHAPES)]
317186
fn infallible_bool<T: BenchPrimitive>(bencher: Bencher, &shape: &InputShape) {
@@ -346,18 +215,6 @@ fn deferred_i64(bencher: Bencher, &shape: &InputShape) {
346215
bench_row_fn(bencher, &DeferredI64, make_args::<i64>(shape));
347216
}
348217

349-
#[vortex_bench_support::cpu_features]
350-
#[divan::bench(args = VALIDITY_PATTERNS)]
351-
fn filtered_owned_i64(bencher: Bencher, &validity: &ValidityPattern) {
352-
bench_row_fn(bencher, &FilteredOwnedI64, make_filtered_args(validity));
353-
}
354-
355-
#[vortex_bench_support::cpu_features]
356-
#[divan::bench(args = VALIDITY_PATTERNS)]
357-
fn filtered_sink_i64(bencher: Bencher, &validity: &ValidityPattern) {
358-
bench_row_fn(bencher, &FilteredSinkI64, make_filtered_args(validity));
359-
}
360-
361218
fn make_args<T: BenchPrimitive>(shape: InputShape) -> VecExecutionArgs {
362219
let args = match shape {
363220
InputShape::PerRowPerRow => vec![T::per_row(0), T::per_row(1)],
@@ -368,16 +225,6 @@ fn make_args<T: BenchPrimitive>(shape: InputShape) -> VecExecutionArgs {
368225
VecExecutionArgs::new(args, ROWS)
369226
}
370227

371-
fn make_filtered_args(validity: ValidityPattern) -> VecExecutionArgs {
372-
let values = PrimitiveArray::new(
373-
(0..ROWS).map(|index| index as i64).collect::<Buffer<_>>(),
374-
Validity::from_iter((0..ROWS).map(|index| validity.is_valid(index))),
375-
)
376-
.into_array();
377-
378-
VecExecutionArgs::new(vec![values], ROWS)
379-
}
380-
381228
fn bench_row_fn<F: RowFn<Options = EmptyOptions>>(
382229
bencher: Bencher,
383230
function: &F,

‎vortex-array/benches/row_fn_utf8.rs‎

Lines changed: 0 additions & 77 deletions
This file was deleted.

‎vortex-array/benches/take_chunked.rs‎

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,6 @@ const NESTED_WIDTH: usize = 8;
3939
enum Pattern {
4040
Sorted,
4141
Shuffled,
42-
Repeated,
4342
Duplicate90,
4443
Duplicate99,
4544
SameChunk,
@@ -52,7 +51,6 @@ impl Display for Pattern {
5251
f.write_str(match self {
5352
Self::Sorted => "sorted",
5453
Self::Shuffled => "shuffled",
55-
Self::Repeated => "repeated",
5654
Self::Duplicate90 => "duplicate90",
5755
Self::Duplicate99 => "duplicate99",
5856
Self::SameChunk => "same_chunk",
@@ -157,11 +155,10 @@ fn cases() -> Vec<Case> {
157155
}
158156
}
159157

160-
for pattern in [
161-
Pattern::Repeated,
162-
Pattern::Duplicate90,
163-
Pattern::Duplicate99,
164-
] {
158+
// Duplicate90 and Duplicate99 cover heavy duplication. An all-identical index vector used
159+
// to sit here too, but it reported either 24 µs or 104 µs on pull requests that changed no
160+
// Vortex code, so the degenerate case measured nothing useful.
161+
for pattern in [Pattern::Duplicate90, Pattern::Duplicate99] {
165162
cases.push(Case::new("duplicates", pattern, 16, 1_000));
166163
}
167164

@@ -259,8 +256,7 @@ fn index_values(case: Case) -> Vec<u64> {
259256
let mut indices = match case.pattern {
260257
Pattern::Sorted | Pattern::Shuffled => (0..case.indices)
261258
.map(|_| advance_random(&mut state) % u64::try_from(SOURCE_LEN).unwrap())
262-
.collect(),
263-
Pattern::Repeated => vec![u64::try_from(SOURCE_LEN / 2).unwrap(); case.indices],
259+
.collect::<Vec<_>>(),
264260
Pattern::Duplicate90 | Pattern::Duplicate99 => {
265261
let divisor = match case.pattern {
266262
Pattern::Duplicate90 => 10,

0 commit comments

Comments
 (0)