Skip to content

bench: run JOB in post-commit develop benchmarks - #11095

Merged
joseph-isaacs merged 1 commit into
developfrom
ji/develop-bench-job
Nov 2, 2026
Merged

joseph-isaacs merged 1 commit into
developfrom
ji/develop-bench-job

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

JOB on NVMe (DuckDB only) previously ran only in the pr-full / pr-all SQL presets, so it never reached the benchmarks website. This adds it to the develop preset, so post-commit runs ingest its results to the benchmarks database like the other SQL suites.

JOB on NVMe (DuckDB only) previously ran only in the pr-full and pr-all
presets. It takes about 7 minutes and runs in parallel with the rest of
the SQL matrix, so it does not extend the develop run. Running it on
develop ingests its results to the benchmarks database for the website.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01UeniBCuWZb3y6Lu63GcpqW
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 2, 2026 12:59
@codspeed

codspeed Bot commented Nov 2, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 10.89%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

❌ 1 regressed benchmark
✅ 2317 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime starts_with_avx2[64, VarBin] 53.8 µs 60.4 µs -10.89%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/develop-bench-job (44bdeff) with develop (7c9bb58)

Open in CodSpeed

@joseph-isaacs
joseph-isaacs merged commit 75a4b92 into develop Nov 2, 2026
103 of 105 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/develop-bench-job branch November 2, 2026 13:54
connortsui20 added a commit that referenced this pull request Nov 2, 2026
Short walltime benchmarks can report changes on unchanged code because
CodSpeed compares the fastest sample. Batch the bitpacked filter and
take samples, and keep unreliable cases local. Exclusions live beside
the benchmark definitions, using `#[cpu_features(exclude = ["neon"])]`
for whole CPU variants and source-level selection for individual inputs.
CI commands stay generic.

Move the chunked-take family to walltime with 16 iterations per sample
so it measures the operation's allocator and syscall costs. Keep its x86
variants local after repeatability checks failed; all 27 NEON cases
remain in CI. The guide retains practical examples and measurement
guidance without an exclusion catalog or a repeatability tool.

<details>
<summary>Measurement evidence</summary>

Spread is `max / min - 1` across independent processes, with 1000 Divan
samples per process and ten processes on each of three matching hosts.

| Case | Before | Fixed sampling |
| --- | --- | --- |
| AVX2 `filter[80]`, sample size 64 | 9.36% | 1.45% |
| AVX-512 `threshold[80]`, sample size 16 | 8.26% | 1.69% |

Across independent base and head AVX2 builds, the fixed filter's worst
spread was 4.66% over 60 processes. AVX-512 filter cases stayed within
1.78%. The excluded NEON string case still varied by 6.38%.

Measurements used existing CI executables with matching sample-size
overrides from [run
41449104113](https://github.com/vortex-data/vortex/actions/runs/41449104113)
(`44bdefff6e69`) and [run
41442182591](https://github.com/vortex-data/vortex/actions/runs/41442182591)
(`7c9bb58e39fe`), on three `c7i.metal-24xl` hosts for x86 and three
`c7g.metal` hosts for NEON. These measure repeatability, not kernel
speedups.

Other Rust exclusions follow unexplained variation recorded in
[#10808](#10808) and repeated
on the workflow-only
[#11095](#11095 (comment)).
The two Python integer scalar scans include variable calling-thread
reactor polling, visible in
[#11102](#11102), even with
memory mapping and background workers disabled.

The chunked-take warning on `dfb32c5`, compared with `d8e7467`, came
from mimalloc lazily allocating page-map metadata. The affected
`small_m/shuffled/primitive/nonnull/chunks=2048/indices=16` case made
one `mmap` and one `prctl` call totaling 23.3 µs; the base trace made
neither. An untimed warm-up removed the syscall warning but produced
eight regression alerts with unchanged kernel code under
allocation-excluded simulation. Walltime measures elapsed time without
that allocation exclusion and establishes separate per-CPU baselines.

Walltime repeats used the CI executables from `29920deef0`, 1000 samples
per process, and ten independent processes on each of three matching
hosts per architecture. All 27 NEON cases stayed within 4.35% spread;
the originally flagged case stayed within 2.88%. The worst x86 spreads
were 11.04% for AVX2 and 11.20% for AVX-512. Increasing the sample size
to 64 on two troublesome x86 cases still produced 12.15% and 13.38%
spread, so those CPU variants remain local through the macro exclusion.

</details>

Focused native and simulation Clippy checks passed. Test-mode runs
confirmed the local, AVX2, and NEON string case selection and the
macro's NEON exclusion. Pinned-nightly formatting, Ruff, ty, YAML
linting, and patch whitespace checks passed. All 66 CI checks passed on
`29920deef0`, including CodSpeed analysis with zero syscall warnings.
The final x86 exclusion passed focused native and simulation Clippy
checks and test-mode checks for both excluded variants. CI is rerunning
for that final change. Local Git hooks were bypassed.

AI assistance: Codex helped investigate, implement and prepare this PR.

---------

Signed-off-by: Connor Tsui <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants