Skip to content

bench: resize the microbenchmarks that flip on unrelated pull requests - #9951

Merged
connortsui20 merged 1 commit into
developfrom
ct/beautiful-maxwell-7eyyvp
Sep 21, 2026
Merged

connortsui20 merged 1 commit into
developfrom
ct/beautiful-maxwell-7eyyvp

Conversation

@connortsui20

@connortsui20 connortsui20 commented Sep 20, 2026 •

Copy link
Copy Markdown
Member

Summary

CodSpeed has been flagging regressions and improvements on pull requests that change no Rust code, such as Renovate digest bumps and docs edits. I collected the CodSpeed report of every pull request merged since 2026-08-24 (195 reports, 1201 flagged rows, 271 distinct benchmarks) and cross-referenced each flagged benchmark with the crates the pull request touched, using the workspace dependency graph. The benchmarks in this PR flipped repeatedly on pull requests that could not have affected them, and the common cause is their size.

Two patterns:

  • In the simulation shards CodSpeed runs a benchmark once and adds a fixed cost of roughly half a microsecond of reported time. A benchmark whose real work is tens of nanoseconds reports mostly that floor, and the floor differs between runs. allocate_drop_bytes[0] does about 3 ns of work locally, is reported as 0.5 µs, and flipped on 37 pull requests.
  • On the walltime legs, iterations of a few microseconds flip by 10% to 90% between runs of identical code, while longer iterations in the same files do not.

This is the first of three stacked PRs. The second removes third-party and frozen baselines from CodSpeed. The third gives every benchmark binary the same allocator.

Changes

Benchmark Flips on unrelated PRs (all PRs) Change
allocation.rs: allocate_drop_*[0] 27 (46), 27 (37), 8 (11), 8 (11) Drop the zero-byte size, which allocates nothing, and repeat each allocate-and-drop pair 256 times per iteration (the from_vec cases copy, so their batch shrinks with the size to about a mebibyte per iteration)
filter_fixed_width.rs: random_i8, random_i16, cached_indices_* 22 (44), 11 (23), 8 (23), 8 (9), 4 (9), 4 (6) Size inputs by bytes (64 KiB) instead of 4096 elements. The i8 case did about 1 µs of real work and was reported at 67 to 91 µs
run_end_compress.rs: decompress[u64, (4000, 1024)] 27 (47) Drop this grid point. (10000, 1024) and (10000, 4096) keep long runs covered
varbinview_compact.rs: compact*[(_, 90)] 0 (10, all vortex-array PRs) Drop the 90% utilization cases: there is nothing to compact, so they did 14 ns of work
binary_ops.rs: mul_u32_nonnull, mul_i32_nonnull, *_shapes[(128, _)] 31 (57), 10 (19), 3 (9), 3 (8), 2 (6) 256 KiB per input instead of 96 KiB, and no 128-row shape case (2 µs per iteration, swings up to 95%)
collect_bool.rs: words_gather_dispatch[1024] 7 (9), 3 (5) Gather sizes 64K and 1M instead of 1K and 64K. The 1K case ran in tens of nanoseconds
lane_kernels.rs: lanezip_checked_add_u32 1 (2) 65536 lanes on the walltime legs instead of 16384. The casts keep 16384 in simulation
take_primitive.rs: dict_canonicalize_gt_u8[16000000] 4 (15) Drop the 16M case. It ran for 7 to 11 ms, over the 1 ms budget, and swung by up to 60%
zone_map_prune.rs: or_chain 1 (1) 4-term chain instead of 16. The 16-term chain ran for 1.4 ms at 1024 zones, over the budget, and longer at 8192 zones

The benchmarking guide gets a section on the floor: how small is too small, and what to do about it.

The CodSpeed report on this PR will show a large change for every resized benchmark, and the dropped cases will show as skipped until they are archived. Whether the flips stop can only be confirmed over the coming pull requests.

Local per-iteration medians after the change (4 vCPU VM, plain cargo bench)
Benchmark Before After
allocation: allocate_drop_* 3 to 70 ns 3 to 10 µs
allocation: from_vec_drop_* 30 ns to 1.4 µs 4 to 23 µs
filter_fixed_width: random_i8[0.5] 1.0 µs 7.0 µs
filter_fixed_width: random_i256[0.8] (4096 elements) 4.1 µs
collect_bool: words_gather_dispatch 23 ns (1K), 1.7 µs (64K) 1.7 µs (64K), 26 µs (1M)
collect_bool: words_gather_scalar 0.4 µs (1K), 25 µs (64K) 24 µs (64K), 420 µs (1M)
lane_kernels: lanezip_checked_add_u32 12.9 µs 56 µs
binary_ops: mul_u32_nonnull (96 KiB inputs) 22.6 µs
binary_ops: add_shapes[(32768, PerRowPerRow)] 13.5 µs
take_primitive: dict_canonicalize_gt_u8[1000000] 654 µs unchanged
zone_map_prune: or_chain (16 / 1024 / 8192 zones) 133 / 171 / 360 µs 30 / 33 / 79 µs
run_end_compress: decompress[u64, _] 0.6 to 8 µs unchanged (one point dropped)
varbinview_compact: compact*[(_, 10)] 10 to 33 µs unchanged (90% cases dropped)

CodSpeed's simulation reports roughly ten times these numbers for medium-sized cases, so everything stays well under 1 ms.

Checks run:

  • cargo clippy --locked --benches -p vortex-buffer -p vortex-compute -p vortex-runend -p vortex-layout -p vortex-array --features vortex-array/_test-harness -- -D warnings
  • cargo +nightly-2026-09-10 fmt on the touched crates
  • Every resized benchmark was run locally with --bench (table above)

Not run: the CodSpeed jobs themselves; they run on this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn


Generated by Claude Code

@codspeed

codspeed Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 80.81%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 11 improved benchmarks
❌ 131 regressed benchmarks
✅ 2045 untouched benchmarks
🆕 30 new benchmarks
⏩ 284 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation from_vec_drop_arrow[256] 2.8 µs 133.9 µs -97.9%
❌ Simulation from_vec_drop_arrow[1024] 2.9 µs 136.1 µs -97.89%
❌ Simulation from_vec_drop_vortex[256] 3.1 µs 142.3 µs -97.84%
❌ Simulation from_vec_drop_vortex[1024] 3.1 µs 144.8 µs -97.84%
❌ Simulation from_vec_drop_bytes[1024] 2.2 µs 99.4 µs -97.8%
❌ Simulation from_vec_drop_bytes[256] 2.2 µs 96.5 µs -97.74%
❌ Simulation from_vec_drop_vortex[64] 3 µs 114.7 µs -97.37%
❌ Simulation from_vec_drop_arrow[64] 2.8 µs 105.3 µs -97.33%
❌ Simulation allocate_freeze_drop_vortex_minimal_alignment[64] 3.3 µs 109.1 µs -96.95%
❌ Simulation allocate_freeze_drop_vortex_custom[64] 4.2 µs 135.6 µs -96.91%
❌ Simulation allocate_freeze_drop_vortex[64] 3.4 µs 109.3 µs -96.86%
❌ Simulation from_vec_drop_bytes[64] 2.2 µs 68.6 µs -96.82%
❌ Simulation allocate_freeze_drop_vortex[256] 3.5 µs 109.4 µs -96.81%
❌ Simulation allocate_freeze_drop_bytes[64] 3.2 µs 97 µs -96.74%
❌ Simulation allocate_freeze_drop_bytes[256] 3.2 µs 97.1 µs -96.68%
❌ Simulation allocate_freeze_drop_bytes[1024] 3.3 µs 97.1 µs -96.63%
❌ Simulation allocate_freeze_drop_arrow[64] 3.8 µs 104.9 µs -96.38%
❌ Simulation allocate_freeze_drop_vortex_minimal_alignment[65536] 6 µs 148.5 µs -95.99%
❌ Simulation allocate_freeze_drop_vortex_minimal_alignment[16384] 6 µs 148.3 µs -95.99%
❌ Simulation allocate_freeze_drop_vortex[1024] 6 µs 148.6 µs -95.96%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/beautiful-maxwell-7eyyvp (d99db51) with develop (709ef16)

Open in CodSpeed

Footnotes

  1. 284 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Comment thread vortex-buffer/benches/allocation.rs Resolved
CodSpeed runs each benchmark once and adds roughly half a microsecond of
fixed cost, so a benchmark doing tens of nanoseconds of work reported that
floor and moved by more than 10% on pull requests that touched no Rust.
The walltime legs have the same problem at the microsecond scale.

- allocation: drop the zero-byte size and repeat each allocate-and-drop
  pair 256 times per iteration, fewer for the copying from_vec cases.
- varbinview_compact: drop the 90% utilization cases, which have nothing
  to compact.
- filter_fixed_width: size inputs by bytes (64 KiB) instead of 4096
  elements, so narrow widths do measurable work.
- run_end_compress: drop the (4000, 1024) grid point.
- collect_bool, lane_kernels, binary_ops: longer inputs on the walltime
  legs, and no 128-row shape case.
- take_primitive: drop the 16M dictionary case, which ran for 7 to 11 ms.
- zone_map_prune: a 4-term OR chain instead of 16, which ran for 1.4 ms.

Document the floor in the benchmarking guide.

Signed-off-by: Claude <[email protected]>
@connortsui20
connortsui20 force-pushed the ct/beautiful-maxwell-7eyyvp branch from 8442a96 to d99db51 Compare September 21, 2026 18:23
@connortsui20
connortsui20 merged commit 5c7f370 into develop Sep 21, 2026
121 of 123 checks passed
@connortsui20
connortsui20 deleted the ct/beautiful-maxwell-7eyyvp branch September 21, 2026 18:49
connortsui20 added a commit that referenced this pull request Sep 22, 2026
…9994)

## Summary

Mimalloc's deferred page collection and arena purging can land inside a
single-shot benchmark and get reported as a Vortex regression. In [this
CI
comparison](https://app.codspeed.io/vortex-data/vortex/runs/compare/6ab29cf078100e55aedd4885..6ab2a14f78100e55aedd4921),
`_mi_theap_collect_retired` and its callees account for an additional
22.30 µs in the dictionary case and 38.64 µs in the FSL case, closely
matching their reported gaps.

## Changes

Enable `exclude-allocations` for the array dictionary and selection
simulation shards, which contain `dict_compress`, `take_fsl`, and
`take_chunked`. This keeps mimalloc in use while excluding allocator
time from these scores. Allocation-focused benchmarks in the core shard
continue to measure allocator cost.

<details>
<summary>Previous attempts</summary>

- #8742 and #8861 addressed earlier two-level CodSpeed results by adding
mimalloc. #8807 tracked the cases that remained flaky.
- #9951 resized noisy cases, and #9953 extended mimalloc to the
remaining benchmark binaries. Deferred cleanup still varies within
mimalloc.
- #9174 included a commit titled “Try and exclude codspeed allocations,”
but its diff only upgraded the action. Its report used runner 5.0.1,
where [allocation exclusion was already disabled by
default](CodSpeedHQ/codspeed#489). No explicit
opt-in was added. This PR sets it for the affected shards.

</details>

Signed-off-by: Connor Tsui <[email protected]>
connortsui20 added a commit that referenced this pull request Sep 22, 2026
…9980)

## Summary

The decimal kernel benchmarks landed with a 1,024 row case. For `i128`
that runs in 0.4 to 1.2 µs
on the walltime legs, below what those machines resolve, so it reports
mostly jitter: it moved by
10 to 19% on ten of the last thirteen CodSpeed reports, every one of
them on a pull request that
touched no decimal code. No 8,192 row case and no `i256` case has moved
in those reports.

## Changes

Derive each kernel case's row count from a working set of 256 KiB and 1
MiB of input plus output,
rather than fixing it at 1,024 and 8,192 rows:

| Decimal width | Small case | Large case |
| --- | ---: | ---: |
| `i128` | 8,192 rows | 32,768 rows |
| `i256` | 4,096 rows | 16,384 rows |

Both widths now do comparable work, every case takes several
microseconds instead of fractions of
one, and the largest still fits the 1 MiB L2 cache of the Graviton leg,
so these stay measurements
of kernel code rather than of memory bandwidth.

The array benchmarks in the same files are `#[cfg(not(codspeed))]` and
never run in CI, so they
keep their sizes for quick local runs. The benchmarking guide gains the
measured walltime floor
next to the simulation floor documented in #9951.

Expect a one-time batch of CodSpeed rows for these eight cases when this
merges, since the inputs
change size.

Checked with `cargo clippy -p vortex-decimal-byte-parts --all-targets
--all-features -- -D warnings`
and `cargo fmt`. No benchmarks were run locally: this session is a
shared cloud VM, so its walltime
numbers would not mean anything.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn


---
_Generated by [Claude Code](https://claude.ai/code)_

---------

Signed-off-by: Connor Tsui <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants