Repository navigation
fix(metrics): count num_scalings without the off-by-one from shift(-1) - #139
Anna Pavlenko (apavlen) merged 2 commits into
Conversation
calculate_metrics counted CURR_LIMIT changes with shift(-1), which leaves the last row NaN. Since NaN != x is always True in pandas, the final row was always counted as a scaling event, inflating num_scalings by exactly 1 on every run. This skewed ParetoFrontier normalization and ranking, not just the displayed value. Count against shift() and subtract the single phantom leading change instead; the empty-input guard above keeps this from going negative. Fixing the off-by-one also makes a latent crash reachable: preprocess_df built num_scalings_norm only when num_scalings.max() > 0 but filtered on it unconditionally, so a sweep where nothing scales (now possible) hit a KeyError. Move the percentile filter inside the guard. Add fast unit tests for the count, the empty-input guard, and the no-scaling preprocess_df path, and correct four e2e expectations that hardcoded the inflated counts. Fixes microsoft#136
b78e233 to
be1c4ed
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #139 +/- ##
==========================================
+ Coverage 78.19% 78.28% +0.09%
==========================================
Files 23 23
Lines 1087 1087
Branches 109 109
==========================================
+ Hits 850 851 +1
Misses 187 187
+ Partials 50 49 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The localized fixes address both regressions with appropriate unit and end-to-end coverage.
Review effort: Balanced
Findings: None
What changed in this PR
Corrects scaling-event metrics and safely handles configurations with no scaling.
Changes:
- Fixes the
num_scalingsoff-by-one error. - Guards Pareto filtering when all scaling counts are zero.
- Adds regression coverage and updates end-to-end expectations.
| File | Description |
|---|---|
| tests/test_plot_utils_metrics.py | Adds targeted metric and Pareto regression tests. |
| tests/test_e2e_single_run_sim.py | Corrects single-run scaling expectations. |
| tests/test_e2e_multi_run_tune_with_strategy.py | Corrects tuning scaling expectations. |
| src/vasim/simulator/analysis/plot_utils.py | Counts only actual consecutive limit changes. |
| src/vasim/simulator/analysis/ParetoFrontier.py | Skips scaling normalization filtering when all counts are zero. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Friendly ping — this fixes #136 (a |
Fixes #136.
Problem
calculate_metricsinsrc/vasim/simulator/analysis/plot_utils.pycounted scaling events with a backward shift:shift(-1)leaves the last elementNaN, andNaN != xis alwaysTruein pandas, so the final row is always counted as a scaling event —num_scalingsis inflated by exactly 1 on every run (a configuration that never scales reports 1).This isn't only a displayed value:
num_scalingsfeedsParetoFrontiernormalization (num_scalings / num_scalings.max()) and the Pareto ranking, so tuning comparisons between configurations are skewed by it.Fix
plot_utils.py— count againstshift()and subtract the single phantom leading change (the first row has no predecessor, soNaN != valueis alwaysTrue). The existing empty-input guard means this never goes negative.ParetoFrontier.preprocess_df—num_scalings_normwas created only inside thenum_scalings.max() > 0guard but used unconditionally on the next line. Today that line is unreachable because of the off-by-one (num_scalings was never 0); fixing the count makes it reachable for any sweep where no configuration scales, which would raiseKeyError. The percentile filter is moved inside the guard.Tests
tests/test_plot_utils_metrics.pywith fast, deterministic unit tests: the change-count truth table from the issue, the empty-input guard, and the no-scalingpreprocess_dfpath (regression for theKeyError).466→465,347→346,109→108,133→132), each confirmed by running the simulator.Validation
black,isort,flake8, andpylintpass on the changed files.