Repository navigation
Commit 24cd0ea
authored
Exclude remaining noisy CodSpeed cases from CI (#11176)
## Summary
Follow-up to #11110. Keep the remaining noisy cases out of automatic
CodSpeed runs while preserving them for local benchmarks.
The Python scalar exclusions in #11110 never took effect because the
runner exports simulation as `instrumentation`, while the fixture
checked for `simulation`. Use the plugin's resolved mode so these cases
are actually skipped.
## Changes
- Allow a trailing comma in `#[divan::bench(...)]` when the CPU-feature
macro appends its options. The multiline bitpacked take definition
covers this case in CI builds.
- Share the resolved simulation check between the scalar and filter
fixtures. Also skip the 2 integer one-shot filtered scans that still
report system calls under simulation.
- Keep AVX2 bitpacked take with 80 selections local after batching still
produced a regression alert. Retain the 64-selection case and both
AVX-512 cases in CI.
- Keep AVX-512 nullable integer comparison local because its minimum
changes by more than 10% while its median stays stable.
- Keep the small random power-law boolean filter and the large
correlated power-law boolean filter local under simulation. The former
has repeated unexplained changes, and the latter repeatedly makes system
calls that simulation cannot measure.
<details>
<summary>Evidence from recent PR reports</summary>
Reviewed 40 CodSpeed report comments updated between November 2 at 19:28
UTC and November 3 at 19:28 UTC, including their comparison revisions
and the affected measurement modes.
| Case | Evidence |
| --- | --- |
| Python integer scalar scans |
[#11154](#11154 (comment))
still reports the repeated scan's 10 system calls. [A develop run after
#11110](https://github.com/vortex-data/vortex/actions/runs/41677647530/job/126062463160)
executes both integer scalar cases instead of skipping them. The [runner
exports the legacy `instrumentation`
value](https://github.com/CodSpeedHQ/codspeed/blob/main/crates/runner-shared/src/measurement_mode.rs),
which pytest-codspeed 5.0.3 normalizes to simulation. |
| Python integer filtered scans |
[#11130](#11130 (comment))
reports 30 system calls totaling 14.3 ms in `test_scan_filter[int]`.
[#11138](#11138 (comment))
reports 30 calls totaling 16.3 ms in
`test_scan_filter_to_arrow_array[int]`. |
| AVX2 `threshold[80]` |
[#11004](#11004 (comment))
compares `f7b5b1d864` with `e59b236d7b`: 2.183 to 2.531 µs, reported as
a 13.75% regression. Both use 1000 samples of 16 iterations. The head
median is 2.556 µs, so this is not just a single outlying minimum. The
PR does not change the fastlanes code or benchmark. |
| AVX-512 `lt_i64_nullable` |
[#11168](#11168 (comment))
reports 7.342 to 8.217 µs. Several unrelated CUDA PRs report the
opposite direction, including
[#10872](#10872 (comment)),
at 8.208 to 7.348 µs. Their head medians remain around 8.3 to 8.5 µs. |
| Boolean `filter_powerlaw_by_random[10000]` |
[#11167](#11167 (comment))
reports a 16.2% regression,
[#11168](#11168 (comment))
a 23.56% improvement, and
[#11169](#11169 (comment))
a 13.06% regression. The cause is unknown. |
| Boolean `filter_powerlaw_by_correlated_runs[100000]` |
[#10871](#10871 (comment))
and
[#11165](#11165 (comment))
each report 2 system calls totaling about 44 µs. |
The NEON chunked-take alerts on #11090 are on a PR that changes
execution substantially, so those cases remain enabled. Other dictionary
simulation alerts need more attribution before excluding them.
</details>
CI passes formatting, Python lint and script tests, Rust lint,
default-feature and all-feature Rust builds, all 3 CPU walltime
benchmark jobs, and Python benchmarks (36 passed, 4 intentionally
skipped). The previously failed simulation shard also passes. Some
remaining CI jobs and CodSpeed performance analysis are still pending.
No local checks were run. The pre-commit and pre-push hooks were
bypassed as requested.
---------
Signed-off-by: Connor Tsui <[email protected]>1 parent ff7f582 commit 24cd0ea
7 files changed
Lines changed: 40 additions & 20 deletions
File tree
- benchmarks/bench-support/src
- encodings/fastlanes/benches
- vortex-array/benches
- vortex-python/benchmark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
| 88 | + | |
88 | 89 | | |
89 | | - | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
90 | 96 | | |
91 | 97 | | |
92 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
322 | 322 | | |
323 | 323 | | |
324 | 324 | | |
325 | | - | |
| 325 | + | |
| 326 | + | |
326 | 327 | | |
327 | 328 | | |
328 | 329 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
296 | 296 | | |
297 | 297 | | |
298 | 298 | | |
299 | | - | |
| 299 | + | |
| 300 | + | |
300 | 301 | | |
301 | 302 | | |
302 | 303 | | |
| |||
316 | 317 | | |
317 | 318 | | |
318 | 319 | | |
319 | | - | |
| 320 | + | |
| 321 | + | |
320 | 322 | | |
321 | 323 | | |
322 | 324 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
8 | 10 | | |
9 | 11 | | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
13 | 22 | | |
14 | 23 | | |
15 | 24 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
12 | 20 | | |
13 | | - | |
14 | | - | |
| 21 | + | |
| 22 | + | |
15 | 23 | | |
16 | 24 | | |
17 | 25 | | |
| |||
21 | 29 | | |
22 | 30 | | |
23 | 31 | | |
24 | | - | |
25 | | - | |
| 32 | + | |
| 33 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
6 | 4 | | |
7 | 5 | | |
8 | 6 | | |
| |||
11 | 9 | | |
12 | 10 | | |
13 | 11 | | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
| 12 | + | |
19 | 13 | | |
20 | | - | |
| 14 | + | |
21 | 15 | | |
22 | 16 | | |
23 | 17 | | |
| |||
0 commit comments