Repository navigation
bench: resize the microbenchmarks that flip on unrelated pull requests - #9951
Conversation
Merging this PR will degrade performance by 80.81%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | from_vec_drop_arrow[256] |
2.8 µs | 133.9 µs | -97.9% |
| ❌ | Simulation | from_vec_drop_arrow[1024] |
2.9 µs | 136.1 µs | -97.89% |
| ❌ | Simulation | from_vec_drop_vortex[256] |
3.1 µs | 142.3 µs | -97.84% |
| ❌ | Simulation | from_vec_drop_vortex[1024] |
3.1 µs | 144.8 µs | -97.84% |
| ❌ | Simulation | from_vec_drop_bytes[1024] |
2.2 µs | 99.4 µs | -97.8% |
| ❌ | Simulation | from_vec_drop_bytes[256] |
2.2 µs | 96.5 µs | -97.74% |
| ❌ | Simulation | from_vec_drop_vortex[64] |
3 µs | 114.7 µs | -97.37% |
| ❌ | Simulation | from_vec_drop_arrow[64] |
2.8 µs | 105.3 µs | -97.33% |
| ❌ | Simulation | allocate_freeze_drop_vortex_minimal_alignment[64] |
3.3 µs | 109.1 µs | -96.95% |
| ❌ | Simulation | allocate_freeze_drop_vortex_custom[64] |
4.2 µs | 135.6 µs | -96.91% |
| ❌ | Simulation | allocate_freeze_drop_vortex[64] |
3.4 µs | 109.3 µs | -96.86% |
| ❌ | Simulation | from_vec_drop_bytes[64] |
2.2 µs | 68.6 µs | -96.82% |
| ❌ | Simulation | allocate_freeze_drop_vortex[256] |
3.5 µs | 109.4 µs | -96.81% |
| ❌ | Simulation | allocate_freeze_drop_bytes[64] |
3.2 µs | 97 µs | -96.74% |
| ❌ | Simulation | allocate_freeze_drop_bytes[256] |
3.2 µs | 97.1 µs | -96.68% |
| ❌ | Simulation | allocate_freeze_drop_bytes[1024] |
3.3 µs | 97.1 µs | -96.63% |
| ❌ | Simulation | allocate_freeze_drop_arrow[64] |
3.8 µs | 104.9 µs | -96.38% |
| ❌ | Simulation | allocate_freeze_drop_vortex_minimal_alignment[65536] |
6 µs | 148.5 µs | -95.99% |
| ❌ | Simulation | allocate_freeze_drop_vortex_minimal_alignment[16384] |
6 µs | 148.3 µs | -95.99% |
| ❌ | Simulation | allocate_freeze_drop_vortex[1024] |
6 µs | 148.6 µs | -95.96% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/beautiful-maxwell-7eyyvp (d99db51) with develop (709ef16)
Footnotes
-
284 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
CodSpeed runs each benchmark once and adds roughly half a microsecond of fixed cost, so a benchmark doing tens of nanoseconds of work reported that floor and moved by more than 10% on pull requests that touched no Rust. The walltime legs have the same problem at the microsecond scale. - allocation: drop the zero-byte size and repeat each allocate-and-drop pair 256 times per iteration, fewer for the copying from_vec cases. - varbinview_compact: drop the 90% utilization cases, which have nothing to compact. - filter_fixed_width: size inputs by bytes (64 KiB) instead of 4096 elements, so narrow widths do measurable work. - run_end_compress: drop the (4000, 1024) grid point. - collect_bool, lane_kernels, binary_ops: longer inputs on the walltime legs, and no 128-row shape case. - take_primitive: drop the 16M dictionary case, which ran for 7 to 11 ms. - zone_map_prune: a 4-term OR chain instead of 16, which ran for 1.4 ms. Document the floor in the benchmarking guide. Signed-off-by: Claude <[email protected]>
8442a96 to
d99db51
Compare
…9994) ## Summary Mimalloc's deferred page collection and arena purging can land inside a single-shot benchmark and get reported as a Vortex regression. In [this CI comparison](https://app.codspeed.io/vortex-data/vortex/runs/compare/6ab29cf078100e55aedd4885..6ab2a14f78100e55aedd4921), `_mi_theap_collect_retired` and its callees account for an additional 22.30 µs in the dictionary case and 38.64 µs in the FSL case, closely matching their reported gaps. ## Changes Enable `exclude-allocations` for the array dictionary and selection simulation shards, which contain `dict_compress`, `take_fsl`, and `take_chunked`. This keeps mimalloc in use while excluding allocator time from these scores. Allocation-focused benchmarks in the core shard continue to measure allocator cost. <details> <summary>Previous attempts</summary> - #8742 and #8861 addressed earlier two-level CodSpeed results by adding mimalloc. #8807 tracked the cases that remained flaky. - #9951 resized noisy cases, and #9953 extended mimalloc to the remaining benchmark binaries. Deferred cleanup still varies within mimalloc. - #9174 included a commit titled “Try and exclude codspeed allocations,” but its diff only upgraded the action. Its report used runner 5.0.1, where [allocation exclusion was already disabled by default](CodSpeedHQ/codspeed#489). No explicit opt-in was added. This PR sets it for the affected shards. </details> Signed-off-by: Connor Tsui <[email protected]>
…9980) ## Summary The decimal kernel benchmarks landed with a 1,024 row case. For `i128` that runs in 0.4 to 1.2 µs on the walltime legs, below what those machines resolve, so it reports mostly jitter: it moved by 10 to 19% on ten of the last thirteen CodSpeed reports, every one of them on a pull request that touched no decimal code. No 8,192 row case and no `i256` case has moved in those reports. ## Changes Derive each kernel case's row count from a working set of 256 KiB and 1 MiB of input plus output, rather than fixing it at 1,024 and 8,192 rows: | Decimal width | Small case | Large case | | --- | ---: | ---: | | `i128` | 8,192 rows | 32,768 rows | | `i256` | 4,096 rows | 16,384 rows | Both widths now do comparable work, every case takes several microseconds instead of fractions of one, and the largest still fits the 1 MiB L2 cache of the Graviton leg, so these stay measurements of kernel code rather than of memory bandwidth. The array benchmarks in the same files are `#[cfg(not(codspeed))]` and never run in CI, so they keep their sizes for quick local runs. The benchmarking guide gains the measured walltime floor next to the simulation floor documented in #9951. Expect a one-time batch of CodSpeed rows for these eight cases when this merges, since the inputs change size. Checked with `cargo clippy -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings` and `cargo fmt`. No benchmarks were run locally: this session is a shared cloud VM, so its walltime numbers would not mean anything. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn --- _Generated by [Claude Code](https://claude.ai/code)_ --------- Signed-off-by: Connor Tsui <[email protected]>
Summary
CodSpeed has been flagging regressions and improvements on pull requests that change no Rust code, such as Renovate digest bumps and docs edits. I collected the CodSpeed report of every pull request merged since 2026-08-24 (195 reports, 1201 flagged rows, 271 distinct benchmarks) and cross-referenced each flagged benchmark with the crates the pull request touched, using the workspace dependency graph. The benchmarks in this PR flipped repeatedly on pull requests that could not have affected them, and the common cause is their size.
Two patterns:
allocate_drop_bytes[0]does about 3 ns of work locally, is reported as 0.5 µs, and flipped on 37 pull requests.This is the first of three stacked PRs. The second removes third-party and frozen baselines from CodSpeed. The third gives every benchmark binary the same allocator.
Changes
allocation.rs:allocate_drop_*[0]from_veccases copy, so their batch shrinks with the size to about a mebibyte per iteration)filter_fixed_width.rs:random_i8,random_i16,cached_indices_*i8case did about 1 µs of real work and was reported at 67 to 91 µsrun_end_compress.rs:decompress[u64, (4000, 1024)](10000, 1024)and(10000, 4096)keep long runs coveredvarbinview_compact.rs:compact*[(_, 90)]binary_ops.rs:mul_u32_nonnull,mul_i32_nonnull,*_shapes[(128, _)]collect_bool.rs:words_gather_dispatch[1024]lane_kernels.rs:lanezip_checked_add_u32take_primitive.rs:dict_canonicalize_gt_u8[16000000]zone_map_prune.rs:or_chainThe benchmarking guide gets a section on the floor: how small is too small, and what to do about it.
The CodSpeed report on this PR will show a large change for every resized benchmark, and the dropped cases will show as skipped until they are archived. Whether the flips stop can only be confirmed over the coming pull requests.
Local per-iteration medians after the change (4 vCPU VM, plain
cargo bench)allocation:allocate_drop_*allocation:from_vec_drop_*filter_fixed_width:random_i8[0.5]filter_fixed_width:random_i256[0.8]collect_bool:words_gather_dispatchcollect_bool:words_gather_scalarlane_kernels:lanezip_checked_add_u32binary_ops:mul_u32_nonnullbinary_ops:add_shapes[(32768, PerRowPerRow)]take_primitive:dict_canonicalize_gt_u8[1000000]zone_map_prune:or_chain(16 / 1024 / 8192 zones)run_end_compress:decompress[u64, _]varbinview_compact:compact*[(_, 10)]CodSpeed's simulation reports roughly ten times these numbers for medium-sized cases, so everything stays well under 1 ms.
Checks run:
cargo clippy --locked --benches -p vortex-buffer -p vortex-compute -p vortex-runend -p vortex-layout -p vortex-array --features vortex-array/_test-harness -- -D warningscargo +nightly-2026-09-10 fmton the touched crates--bench(table above)Not run: the CodSpeed jobs themselves; they run on this PR.
🤖 Generated with Claude Code
https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn
Generated by Claude Code