Repository navigation
CodSpeed: test_pickle_load[arm] is bimodal and randomly flags unrelated PRs #1110
Description
test_pickle_load[arm] (added in #1092) is bimodal: it flips between ~1.6 ms and ~1.8 ms across commits that don't touch the pickle path at all, so with the 2% threshold it randomly paints unrelated PRs red.
Evidence from this week:
- Add callable support for
decimal_formatoption #978 (decimal_format encoder option, head 045a25c vs main b36a4a1): flagged -10.05% (1.6 -> 1.8 ms), check red. The diff cannot affect struct pickling. - Fix
to_builtinswithfrozendict#1105 (to_builtins frozendict fix, head 5a54d59 vs main 8f7ac06): same benchmark reported +10.28% (1.8 -> 1.6 ms), check green (improvements don't gate). - The baseline itself flipped between those two comparisons: main measured 1.6 ms at b36a4a1 (06-27) and 1.8 ms at 8f7ac06 (06-30), with exactly one commit in between (Add
frozendictsupport for 3.15+ #1052, frozendict) that doesn't touch the pickle path either.
Since this is simulation mode (instruction counting), it's not run-to-run measurement noise: the benchmark appears sensitive to code layout/alignment in _core.c, so any code addition can toggle it between the two states. Related to the general variance discussion in #1075 and CodSpeedHQ/codspeed#409.
Options (all live in the CodSpeed dashboard, admin only - there's no repo-side config for this):
- Set a per-benchmark regression threshold for
test_pickle_load[arm]above the flip amplitude (say 15%). Keeps measuring, stops the false gating. - Ignore the benchmark entirely in CodSpeed.
- The repo-wide "informational status check" toggle, which would also address the "CI looks perpetually broken" complaint from Fix crash on incorrect
typing.ClassVarannotation #1097.
Option 1 seems the most targeted. Worth checking whether the sibling test_pickle_* benchmarks from #1092 show the same behavior before picking.
Activity
provinzkraut commentedon Jul 2, 2026
This isn't specific to that benchmark. It's an issue with running the benchmarks in different envs. We need to revisit the continuous benchmarking approach, since codspeed could not delivery on its promises regarding stability.
Siyet commentedon Jul 3, 2026
Might well be both. For the #978 flag specifically the report had no "different runtime environments" warning, and both baseline points (b36a4a1 at 1.6 ms, 8f7ac06 at 1.8 ms) were regular post-#1079 pushes to main, with the flip caused by a commit that doesn't touch the pickle path at all - which looks like code-layout sensitivity of simulation mode rather than env variance.
Either way the remediation is the same: a 2% threshold can't gate benchmarks with a >=10% inherent swing, whether that swing comes from env mismatch or layout effects. Happy to help evaluate options if you want to revisit the setup.
Siyet commentedon Jul 3, 2026
@ofek @jcrist tagging you for visibility: the practical levers here (per-benchmark thresholds, the informational-mode toggle, or ignoring a benchmark) all live in the CodSpeed dashboard and need admin rights there, so this is actionable only by you two. And if we end up revisiting the benchmarking approach more broadly per Janek's comment above, that decision wants your input anyway.