Skip to content

bench: use mimalloc as the global allocator in every benchmark binary - #9953

Merged
connortsui20 merged 1 commit into
developfrom
ct/codspeed-bench-mimalloc
Sep 21, 2026
Merged

bench: use mimalloc as the global allocator in every benchmark binary#9953
connortsui20 merged 1 commit into
developfrom
ct/codspeed-bench-mimalloc

Conversation

@connortsui20

@connortsui20 connortsui20 commented Sep 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Third of three stacked PRs on CodSpeed flakiness. The first resizes benchmarks, the second removes baselines from CodSpeed.

Twenty-two benchmark binaries already use mimalloc as their global allocator, on the advice of the benchmarking guide; the other 81 use the system allocator. Several simulation benchmarks that allocate inside the timed region flipped between two values on pull requests that could not have affected them:

Benchmark Flips on unrelated PRs (all PRs) Max swing
split_collection.rs: cold_misaligned[(16, 64)] 11 (31) 13%
fsst_compress.rs: compress_fsst[(500, 64, 8)] 11 (17) 12%
take_chunked.rs: take[duplicates/repeated/...] 11 (15) 21%
take_primitive.rs: dict_canonicalize_gt_u8[1000000] (avx2 leg) 2 (6) 28%

The system allocator's cost depends on its state, which differs between runner images and between runs; mimalloc does the same work every time. One allocator for every benchmark binary also means no benchmark measures a different allocator from its neighbours.

Changes

  • Every benchmark binary that lacked it gets mimalloc as its global allocator (81 files). The 16 crates that did not depend on it get the workspace mimalloc dev-dependency, so Cargo.lock gains mimalloc in their dependency lists and nothing else.
  • The benchmarking guide's "prefer mimalloc for throughput benchmarks" becomes "every benchmark binary uses mimalloc".
  • The CUDA and DuckDB benchmarks are left alone: they run on different legs and link C++ code.

The CodSpeed report on this PR will show a change for most simulation benchmarks, as expected from swapping the allocator under them. Whether the flips above stop can only be confirmed over the coming pull requests. Two of the flaky benchmarks that already had mimalloc (filter_fixed_width.rs and run_end_compress.rs) needed resizing in the first PR instead, so this removes one variable rather than guaranteeing stability.

Checks run:

  • cargo clippy --locked --benches with -D warnings on every touched crate (vortex-alp, vortex-datetime-parts, vortex-decimal-byte-parts, vortex-fastlanes, vortex-fsst, vortex-onpair, vortex-pco, vortex-runend, vortex-sparse, vortex-array, vortex-arrow, vortex-btrblocks, vortex-buffer, vortex-compressor, vortex-compute, vortex-file, vortex-layout, vortex-mask, vortex)
  • cargo +nightly-2026-09-10 fmt --all and taplo fmt --check on the touched manifests

🤖 Generated with Claude Code

https://claude.ai/code/session_01B6ApyxEXpjiasZbQptsypn

Generated by Claude Code

This was referenced Sep 20, 2026
Merged
Merged
@codspeed

codspeed Bot commented Sep 20, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will regress 96 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 111 improved benchmarks
❌ 96 regressed benchmarks
✅ 2001 untouched benchmarks
⏩ 293 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation collect_bool_u32_gt[1024] 10.5 µs 24.3 µs -56.66%
❌ Simulation preverify_advancing_ptr_unchecked[1000] 11.7 µs 23.9 µs -51%
❌ Simulation execute_mask_then_value[256] 22.1 µs 37.8 µs -41.57%
❌ Simulation sum_v2_i32_nulls_scattered 463.4 µs 741.3 µs -37.49%
❌ Simulation fixed_16_advancing_ptr_safe[100] 19.9 µs 31.7 µs -37.42%
❌ Simulation max_i32_nulls_scattered 499.8 µs 778.3 µs -35.78%
❌ Simulation take_fsl_u32_random[256, 10] 103.4 µs 159.6 µs -35.19%
❌ Simulation decode_primitives[f32, (1000, 512)] 42 µs 64.1 µs -34.4%
❌ Simulation allocate_drop_vortex_minimal_alignment[1024] 63.6 µs 95.3 µs -33.28%
❌ Simulation from_vec_drop_bytes[65536] 10.3 µs 14.6 µs -29.49%
❌ Simulation preverify_advancing_ptr_unchecked[10000] 27 µs 38 µs -28.89%
❌ Simulation take_fsl_f16_force_manual_range_copy[512, 10] 47.5 µs 64.3 µs -26.09%
❌ Simulation bitwise_and_vortex_buffer[1024] 4.5 µs 6 µs -25.04%
❌ Simulation take_fsl_random[128, 10] 105 µs 139.6 µs -24.78%
❌ Simulation from_vec_drop_arrow[65536] 13.1 µs 17.1 µs -23.76%
❌ Simulation from_vec_drop_vortex[65536] 13.7 µs 17.9 µs -23.68%
❌ Simulation take_fsl_f16_force_manual_range_copy[1024, 10] 66.2 µs 84.7 µs -21.87%
❌ Simulation fixed_16_advancing_ptr_safe[1000] 92.5 µs 118.3 µs -21.82%
❌ Simulation canonicalize[16, 32] 52.2 µs 66.7 µs -21.75%
❌ Simulation chunked_dict_primitive_canonical_into[u32, (1000, 10, 10)] 160.3 µs 200.3 µs -19.97%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

Comparing ct/codspeed-bench-mimalloc (f53acf7) with develop (5c7f370)2

Open in CodSpeed

Footnotes

  1. 293 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (e482701) during the generation of this report, so 5c7f370 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@connortsui20
connortsui20 force-pushed the ct/codspeed-bench-mimalloc branch 2 times, most recently from 9f787ec to 2a16543 Compare September 21, 2026 18:49
robert3005
robert3005 approved these changes Sep 21, 2026
View reviewed changes

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do this, I think the actual allocator here doesn't matter as much as predictable allocator

Base automatically changed from ct/codspeed-local-only-baselines to develop September 21, 2026 19:08
@claude @connortsui20
Use mimalloc as the global allocator in every benchmark binary
f53acf7 
Twenty-two benchmark binaries already used mimalloc, on the advice of the
benchmarking guide; the other 81 used the system allocator, whose cost
depends on state that differs between runner images and between runs.
Several simulation benchmarks that allocate inside the timed region
flipped between two values on pull requests that could not have affected
them, among them split_collection's cold_misaligned, fsst_compress's
compress_fsst, and take_chunked's repeated-index case.

Give every benchmark binary the same allocator, add the workspace
mimalloc dev-dependency to the 16 crates that lacked it, and make the
guide's recommendation a rule. The CUDA and DuckDB benchmarks are left
alone: they run on different legs and link C++ code.

Signed-off-by: Claude <[email protected]>
@connortsui20
connortsui20 force-pushed the ct/codspeed-bench-mimalloc branch from 2a16543 to f53acf7 Compare September 21, 2026 19:08
@connortsui20
connortsui20 merged commit f6053a4 into develop Sep 21, 2026
114 of 117 checks passed
@connortsui20
connortsui20 deleted the ct/codspeed-bench-mimalloc branch September 21, 2026 19:30
@connortsui20 connortsui20 mentioned this pull request Sep 22, 2026
Merged
connortsui20 added a commit that referenced this pull request Sep 22, 2026
@connortsui20
ci: exclude allocator time from dictionary and selection benchmarks (#…
36b1f86 
…9994)

## Summary

Mimalloc's deferred page collection and arena purging can land inside a
single-shot benchmark and get reported as a Vortex regression. In [this
CI
comparison](https://app.codspeed.io/vortex-data/vortex/runs/compare/6ab29cf078100e55aedd4885..6ab2a14f78100e55aedd4921),
`_mi_theap_collect_retired` and its callees account for an additional
22.30 µs in the dictionary case and 38.64 µs in the FSL case, closely
matching their reported gaps.

## Changes

Enable `exclude-allocations` for the array dictionary and selection
simulation shards, which contain `dict_compress`, `take_fsl`, and
`take_chunked`. This keeps mimalloc in use while excluding allocator
time from these scores. Allocation-focused benchmarks in the core shard
continue to measure allocator cost.

<details>
<summary>Previous attempts</summary>

- #8742 and #8861 addressed earlier two-level CodSpeed results by adding
mimalloc. #8807 tracked the cases that remained flaky.
- #9951 resized noisy cases, and #9953 extended mimalloc to the
remaining benchmark binaries. Deferred cleanup still varies within
mimalloc.
- #9174 included a commit titled “Try and exclude codspeed allocations,”
but its diff only upgraded the action. Its report used runner 5.0.1,
where [allocation exclusion was already disabled by
default](CodSpeedHQ/codspeed#489). No explicit
opt-in was added. This PR sets it for the affected shards.

</details>

Signed-off-by: Connor Tsui <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robert3005 robert3005 robert3005 approved these changes
@joseph-isaacs joseph-isaacs Awaiting requested review from joseph-isaacs

Assignees

No one assigned

Labels

changelog/ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@connortsui20 @robert3005 @claude