Skip to content

Commit 24cd0ea

Browse files
authored
Exclude remaining noisy CodSpeed cases from CI (#11176)
## Summary Follow-up to #11110. Keep the remaining noisy cases out of automatic CodSpeed runs while preserving them for local benchmarks. The Python scalar exclusions in #11110 never took effect because the runner exports simulation as `instrumentation`, while the fixture checked for `simulation`. Use the plugin's resolved mode so these cases are actually skipped. ## Changes - Allow a trailing comma in `#[divan::bench(...)]` when the CPU-feature macro appends its options. The multiline bitpacked take definition covers this case in CI builds. - Share the resolved simulation check between the scalar and filter fixtures. Also skip the 2 integer one-shot filtered scans that still report system calls under simulation. - Keep AVX2 bitpacked take with 80 selections local after batching still produced a regression alert. Retain the 64-selection case and both AVX-512 cases in CI. - Keep AVX-512 nullable integer comparison local because its minimum changes by more than 10% while its median stays stable. - Keep the small random power-law boolean filter and the large correlated power-law boolean filter local under simulation. The former has repeated unexplained changes, and the latter repeatedly makes system calls that simulation cannot measure. <details> <summary>Evidence from recent PR reports</summary> Reviewed 40 CodSpeed report comments updated between November 2 at 19:28 UTC and November 3 at 19:28 UTC, including their comparison revisions and the affected measurement modes. | Case | Evidence | | --- | --- | | Python integer scalar scans | [#11154](#11154 (comment)) still reports the repeated scan's 10 system calls. [A develop run after #11110](https://github.com/vortex-data/vortex/actions/runs/41677647530/job/126062463160) executes both integer scalar cases instead of skipping them. The [runner exports the legacy `instrumentation` value](https://github.com/CodSpeedHQ/codspeed/blob/main/crates/runner-shared/src/measurement_mode.rs), which pytest-codspeed 5.0.3 normalizes to simulation. | | Python integer filtered scans | [#11130](#11130 (comment)) reports 30 system calls totaling 14.3 ms in `test_scan_filter[int]`. [#11138](#11138 (comment)) reports 30 calls totaling 16.3 ms in `test_scan_filter_to_arrow_array[int]`. | | AVX2 `threshold[80]` | [#11004](#11004 (comment)) compares `f7b5b1d864` with `e59b236d7b`: 2.183 to 2.531 µs, reported as a 13.75% regression. Both use 1000 samples of 16 iterations. The head median is 2.556 µs, so this is not just a single outlying minimum. The PR does not change the fastlanes code or benchmark. | | AVX-512 `lt_i64_nullable` | [#11168](#11168 (comment)) reports 7.342 to 8.217 µs. Several unrelated CUDA PRs report the opposite direction, including [#10872](#10872 (comment)), at 8.208 to 7.348 µs. Their head medians remain around 8.3 to 8.5 µs. | | Boolean `filter_powerlaw_by_random[10000]` | [#11167](#11167 (comment)) reports a 16.2% regression, [#11168](#11168 (comment)) a 23.56% improvement, and [#11169](#11169 (comment)) a 13.06% regression. The cause is unknown. | | Boolean `filter_powerlaw_by_correlated_runs[100000]` | [#10871](#10871 (comment)) and [#11165](#11165 (comment)) each report 2 system calls totaling about 44 µs. | The NEON chunked-take alerts on #11090 are on a PR that changes execution substantially, so those cases remain enabled. Other dictionary simulation alerts need more attribution before excluding them. </details> CI passes formatting, Python lint and script tests, Rust lint, default-feature and all-feature Rust builds, all 3 CPU walltime benchmark jobs, and Python benchmarks (36 passed, 4 intentionally skipped). The previously failed simulation shard also passes. Some remaining CI jobs and CodSpeed performance analysis are still pending. No local checks were run. The pre-commit and pre-push hooks were bypassed as requested. --------- Signed-off-by: Connor Tsui <[email protected]>
1 parent ff7f582 commit 24cd0ea

7 files changed

Lines changed: 40 additions & 20 deletions

File tree

‎benchmarks/bench-support/src/lib.rs‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -106,10 +106,10 @@ pub fn cpu_features(attr: TokenStream, item: TokenStream) -> TokenStream {
106106
// benchmark by leaf name and arguments, so without the suffix every leg would report
107107
// into one `words_gather_dispatch[65536]` series instead of one series each.
108108
let name = function.sig.ident.to_string();
109-
let separator = if existing.is_empty() {
110-
quote!()
111-
} else {
112-
quote!(,)
109+
let separator = match existing.clone().into_iter().last() {
110+
None => quote!(),
111+
Some(proc_macro2::TokenTree::Punct(punct)) if punct.as_char() == ',' => quote!(),
112+
Some(_) => quote!(,),
113113
};
114114
let bench_path = bench.path().clone();
115115
bench.meta = syn::parse_quote! {

‎encodings/fastlanes/benches/bitpacking_take.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,14 @@ fn threshold_fixture(selected_per_chunk: usize) -> (ArrayRef, ArrayRef) {
8585

8686
// Batch short iterations to reduce variation in the fastest walltime sample.
8787
// The NEON cases still vary on unchanged code, so keep those measurements local.
88+
// AVX2 with 80 selections still varies with batching. Retain its 64-selection case in CI.
8889
#[vortex_bench_support::cpu_features(exclude = ["neon"])]
89-
#[divan::bench(args = [64, 80], sample_size = 16)]
90+
#[divan::bench(
91+
args = [64, 80].into_iter().filter(|&count| {
92+
env!("VORTEX_BENCH_VARIANT") != "avx2" || count != 80
93+
}),
94+
sample_size = 16,
95+
)]
9096
fn threshold(bencher: Bencher, selected_per_chunk: usize) {
9197
let (packed, indices) = threshold_fixture(selected_per_chunk);
9298

‎vortex-array/benches/binary_ops.rs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -322,7 +322,8 @@ fn div_decimal_i128_nullable(bencher: Bencher) {
322322
bench_decimal(bencher, lhs, rhs, Operator::Div);
323323
}
324324

325-
#[vortex_bench_support::cpu_features]
325+
// The AVX-512 minimum varies by more than 10% while the median stays stable on unchanged code.
326+
#[vortex_bench_support::cpu_features(exclude = ["avx512"])]
326327
#[divan::bench]
327328
fn lt_i64_nullable(bencher: Bencher) {
328329
let lhs = primitive_nullable(0, 7, I64_LEN).into_array();

‎vortex-array/benches/filter_bool.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -296,7 +296,8 @@ fn filter_powerlaw_by_mostly_false(bencher: Bencher, n: usize) {
296296
});
297297
}
298298

299-
#[divan::bench(args = SIZES)]
299+
// The small case varies in simulation on unrelated PRs. The cause is unknown.
300+
#[divan::bench(args = if cfg!(codspeed) { &[100_000] } else { SIZES })]
300301
fn filter_powerlaw_by_random(bencher: Bencher, n: usize) {
301302
let array = make_power_law_bool_array(n, &mut StdRng::seed_from_u64(ARRAY_SEED));
302303
bencher
@@ -316,7 +317,8 @@ fn filter_powerlaw_by_random(bencher: Bencher, n: usize) {
316317
});
317318
}
318319

319-
#[divan::bench(args = SIZES)]
320+
// The large case repeatedly makes system calls that simulation excludes from its measurement.
321+
#[divan::bench(args = if cfg!(codspeed) { &[10_000] } else { SIZES })]
320322
fn filter_powerlaw_by_correlated_runs(bencher: Bencher, n: usize) {
321323
let array = make_power_law_bool_array(n, &mut StdRng::seed_from_u64(ARRAY_SEED));
322324
bencher

‎vortex-python/benchmark/conftest.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,20 @@
55
from pathlib import Path
66

77
import pytest
8+
from pytest_codspeed.instruments import MeasurementMode
9+
from pytest_codspeed.plugin import get_plugin
810

911
import vortex as vx
1012
from benchmark import data
1113

1214

15+
@pytest.fixture
16+
def codspeed_simulation(pytestconfig: pytest.Config) -> bool:
17+
# Use the resolved mode, including the runner's legacy "instrumentation" alias.
18+
plugin = get_plugin(pytestconfig)
19+
return plugin is not None and plugin.is_codspeed_enabled and plugin.mode == MeasurementMode.Simulation
20+
21+
1322
@pytest.fixture(scope="session", params=list(data.COLUMN_SETS))
1423
def vxf(
1524
tmp_path_factory: pytest.TempPathFactory,

‎vortex-python/benchmark/test_filter.py‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,17 @@
99
from vortex.expr import column
1010

1111

12+
@pytest.fixture
13+
def filter_vxf(request: pytest.FixtureRequest, vxf: vx.VortexFile, codspeed_simulation: bool) -> vx.VortexFile:
14+
assert isinstance(request.node, pytest.Function)
15+
if codspeed_simulation and request.node.callspec.params["vxf"] == "int":
16+
pytest.skip("System calls vary inside these filtered scans under simulation.")
17+
return vxf
18+
19+
1220
@pytest.mark.benchmark(group="filter")
13-
def test_scan_filter(benchmark: BenchmarkFixture, vxf: vx.VortexFile) -> None:
14-
benchmark(lambda: pa.concat_tables(x.to_arrow_table() for x in vxf.scan(expr=column("x") >= 50_000)))
21+
def test_scan_filter(benchmark: BenchmarkFixture, filter_vxf: vx.VortexFile) -> None:
22+
benchmark(lambda: pa.concat_tables(x.to_arrow_table() for x in filter_vxf.scan(expr=column("x") >= 50_000)))
1523

1624

1725
@pytest.mark.benchmark(group="filter")
@@ -21,5 +29,5 @@ def test_repeated_scan_filter(benchmark: BenchmarkFixture, vxf: vx.VortexFile) -
2129

2230

2331
@pytest.mark.benchmark(group="filter")
24-
def test_scan_filter_to_arrow_array(benchmark: BenchmarkFixture, vxf: vx.VortexFile) -> None:
25-
benchmark(lambda: vxf.scan(expr=column("x") >= 50_000).to_arrow_array())
32+
def test_scan_filter_to_arrow_array(benchmark: BenchmarkFixture, filter_vxf: vx.VortexFile) -> None:
33+
benchmark(lambda: filter_vxf.scan(expr=column("x") >= 50_000).to_arrow_array())

‎vortex-python/benchmark/test_scalar_at.py‎

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
11
# SPDX-License-Identifier: Apache-2.0
22
# SPDX-FileCopyrightText: Copyright the Vortex contributors
33

4-
import os
5-
64
import pyarrow as pa
75
import pytest
86
from pytest_benchmark.fixture import BenchmarkFixture
@@ -11,13 +9,9 @@
119

1210

1311
@pytest.fixture
14-
def scalar_vxf(request: pytest.FixtureRequest, vxf: vx.VortexFile) -> vx.VortexFile:
15-
enabled = request.config.getoption("codspeed", default=False) or "CODSPEED_ENV" in os.environ
16-
mode = request.config.getoption("codspeed_mode", default=None)
17-
if mode is None and "CODSPEED_ENV" in os.environ:
18-
mode = os.environ.get("CODSPEED_RUNNER_MODE", "simulation")
12+
def scalar_vxf(request: pytest.FixtureRequest, vxf: vx.VortexFile, codspeed_simulation: bool) -> vx.VortexFile:
1913
assert isinstance(request.node, pytest.Function)
20-
if enabled and mode == "simulation" and request.node.callspec.params["vxf"] == "int":
14+
if codspeed_simulation and request.node.callspec.params["vxf"] == "int":
2115
pytest.skip("Reactor polling varies inside these scalar scans under simulation.")
2216
return vxf
2317

0 commit comments

Comments
 (0)