Skip to content

Commit 6631288

Browse files
committed
fix: bind retained performance reviews to locked dependencies
1 parent 0952b2d commit 6631288

9 files changed

Lines changed: 96 additions & 12 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -360,7 +360,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
360360

361361
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
362362

363-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
363+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
364364

365365
### Synthetic diagnostic scenarios
366366

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
179179

180180
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
181181

182-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
182+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
183183

184184
### Synthetic diagnostic scenarios
185185

‎docs/user-guide/profiles.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
218218

219219
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
220220

221-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
221+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
222222

223223
### Synthetic diagnostic scenarios
224224

‎tests/contracts/test_real_strategy_runner.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -587,3 +587,57 @@ def test_failed_performance_review_does_not_partially_mutate_report() -> None:
587587
)
588588

589589
assert report == before
590+
591+
592+
@pytest.mark.parametrize(
593+
"measured_directly",
594+
[
595+
False,
596+
pytest.param(
597+
True,
598+
marks=pytest.mark.skipif(
599+
sys.platform != "linux", reason="Publication timings identify a Linux host"
600+
),
601+
),
602+
],
603+
)
604+
def test_unreviewed_dependency_change_invalidates_timings(
605+
tmp_path, monkeypatch, measured_directly
606+
) -> None:
607+
benchmark = _load_benchmark()
608+
validation = Path(__file__).parents[2] / "validation"
609+
correctness = json.loads((validation / "REAL_STRATEGY_RESULTS.json").read_text())
610+
report = json.loads((validation / "REAL_STRATEGY_PERFORMANCE.json").read_text())
611+
if measured_directly:
612+
retained = {
613+
(record["case_study"], record["framework"]): record for record in report["records"]
614+
}
615+
616+
def measured_side(*, case_study, framework, side, **_settings):
617+
key = "framework_engine" if side == "framework" else "ml4t_engine"
618+
return copy.deepcopy(retained[(case_study, framework)][key])
619+
620+
monkeypatch.setattr(benchmark, "_measure_side", measured_side)
621+
report = benchmark.build_report(
622+
correctness=correctness, bundle_root=validation, warmups=1, samples=10
623+
)
624+
else:
625+
benchmark.certify_source(
626+
report,
627+
correctness,
628+
validation / "REAL_STRATEGY_RESULTS.json",
629+
"Retain only the reviewed locked dependencies.",
630+
performance_impact="none",
631+
)
632+
assert benchmark.report_failures(report, correctness) == []
633+
634+
lock = tmp_path / "uv.lock"
635+
original = benchmark.DEPENDENCY_PATHS["uv.lock"].read_text()
636+
lock.write_text(original.replace('version = "0.2.0"', 'version = "0.2.1"'))
637+
assert lock.read_text() != original
638+
monkeypatch.setitem(benchmark.DEPENDENCY_PATHS, "uv.lock", lock)
639+
640+
assert any(
641+
"performance-impact review" in failure
642+
for failure in benchmark.report_failures(report, correctness)
643+
)

‎validation/METHODOLOGY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -400,7 +400,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
400400

401401
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
402402

403-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
403+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
404404

405405
### Synthetic diagnostic scenarios
406406

‎validation/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
7676

7777
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
7878

79-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
79+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
8080

8181
### Synthetic diagnostic scenarios
8282

@@ -167,8 +167,8 @@ time a pair that did not pass the retained correctness comparison.
167167
Retain accepted timings when a reviewed change has no material impact on the timed execution.
168168
Record that decision with `real_strategy_benchmark.py --certify-source --performance-impact none
169169
--reason 'TIMED-PATH AND DEPENDENCY REVIEW'`. The review requires current correctness evidence and
170-
binds the exact engine, runners and measurements. It preserves the measurement date, raw samples,
171-
ratios, environment and original correctness reference. Matching numerical outputs alone is not a
170+
binds the exact engine, runners, locked dependencies and measurements. It preserves the measurement
171+
date, raw samples, ratios, environment and original correctness reference. Matching numerical outputs alone is not a
172172
performance-impact assessment. Material or uncertain timing changes require fresh measurements for
173173
the affected execution before updating the performance claims. Unchanged native reference engines
174174
do not need to be rerun solely because Backtest changed; their inputs, settings, dependencies,

‎validation/REAL_STRATEGY_PERFORMANCE.json‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -124,10 +124,19 @@
124124
}
125125
},
126126
{
127-
"certified_at": "2026-10-17T22:23:50.095344+00:00",
127+
"certified_at": "2026-10-17T22:29:51.545630+00:00",
128128
"correctness_evidence_generated_at": "2026-10-17T01:17:00.495296+00:00",
129129
"correctness_evidence_sha256": "ba7f4978cdf675e3d7ff2c998a7eb2f13bc099cb86667d70d56879affdf46a3f",
130130
"correctness_pairs_passed": 17,
131+
"dependencies": {
132+
"uv.lock": "08ffb5f56cf383d8129333aeb0b784cd6f641b44b9afee678a139570a413a344",
133+
"validation/environments/backtrader/uv.lock": "b6716d65024b014103198b1feaec0706009ab4b70ad0ee42f185a576090cd6ae",
134+
"validation/environments/lean/uv.lock": "0af1e08ca7a021310f5afa34aaa724e38b0538ea4c9388f41ae1e2130de0fbc3",
135+
"validation/environments/vectorbt_oss/uv.lock": "cdd7f1882cbf3d283eebafbce4f93f1fcbf4a5eda6f938157e5f141e875c2961",
136+
"validation/environments/vectorbt_pro/uv.lock": "b7ebebc44a571d583ede63f27552594bd7c1b605f5faa9908df30da13be59c38",
137+
"validation/environments/vectorbt_pro_2025_12_31/uv.lock": "e8fed15af7d562ea60b8a320836c6199e098e5392bb4db622e68c8b8d8b6a0a1",
138+
"validation/environments/zipline/uv.lock": "43258cbcc10cdbc29c02da2d41603467b9386ee00f6ea73e501bbffad89d4de0"
139+
},
131140
"engine_source_sha256": "656210f6fe91eb6bac893598909f0363c5951e7bc8f7b14e15d8fb653fb23649",
132141
"measurement_sha256": "4bbc924aeeed357ab44a196c003ba7647c0d0faf60aee25ed534f7481ade1852",
133142
"ml4t_commit": "df10938aa2ae6ed54174d4a065860db196b7c647",
@@ -136,7 +145,7 @@
136145
"reviewed_correctness_sha256": "cbeac90d998cd3aa7830ea4e3e715e03a6437452695d1d7fd99b79f4a8cfb5c2",
137146
"sources": {
138147
"backtrader": "a456cc3bf9ed6708d239f21ed2c2495a3c19a9a8306634fee24619c2c0484459",
139-
"benchmark": "7c592e63020807dc11d965935271331f514780cda31d213fa67758c2413377ea",
148+
"benchmark": "ca9daec5048835a8c5994a30247c070e6fc72bcc0e9d937beceb2d6c0416dbbd",
140149
"comparison_input": "a00bf9383c3e5561333ef0fa734745296cf9cda6fe7155ce950713c1ba2da4f9",
141150
"lean_crypto": "82f1fbb6054fba25d74a4eabbc9ad4eae808caf89c00f560edefa63304da0dc4",
142151
"lean_equity": "f566341d0f4103eda45280684be1b8a91f9da1a46a3a06477cba3b49674d89e8",

‎validation/generate_parity_claims.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -233,8 +233,8 @@ def render_claims(
233233
[
234234
"",
235235
"A performance-impact review retains these measurements for engine source "
236-
f"`{review['engine_source_sha256'][:12]}`. The measured source, review rationale "
237-
"and current correctness evidence are recorded with the raw samples. "
236+
f"`{review['engine_source_sha256'][:12]}`. The measured source, locked dependencies, "
237+
"review rationale and current correctness evidence are recorded with the raw samples. "
238238
"Measurements are repeated when changes materially affect timed execution; "
239239
"a new source revision alone does not require a benchmark rerun.",
240240
]

‎validation/real_strategy_benchmark.py‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,13 @@
3535
**{name: path for name, path in ADAPTER_PATHS.items() if name != "evidence_builder"},
3636
"benchmark": Path(__file__).resolve(),
3737
}
38+
DEPENDENCY_PATHS = {
39+
path.relative_to(PROJECT_ROOT).as_posix(): path
40+
for path in [
41+
PROJECT_ROOT / "uv.lock",
42+
*sorted((VALIDATION_DIR / "environments").glob("*/uv.lock")),
43+
]
44+
}
3845
SUBPROCESS_TIMEOUT_SECONDS = 900
3946
PYTHONS = {
4047
"ml4t": ".venv/bin/python",
@@ -267,6 +274,7 @@ def build_report(
267274
"provenance": {
268275
"ml4t_engine_source_sha256": _tree_digest(PROJECT_ROOT / "src/ml4t/backtest"),
269276
"sources": {name: _sha256(path) for name, path in TIMING_SOURCE_PATHS.items()},
277+
"dependencies": {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()},
270278
},
271279
"correctness_evidence_generated_at": correctness["generated_at"],
272280
"records": records,
@@ -306,7 +314,8 @@ def measurement_digest(report: dict[str, Any]) -> str:
306314
**{key: value for key, value in report.items() if key != "provenance"},
307315
"provenance": {
308316
key: report["provenance"].get(key)
309-
for key in ("ml4t_engine_source_sha256", "sources")
317+
for key in ("ml4t_engine_source_sha256", "sources", "dependencies")
318+
if key in report["provenance"]
310319
},
311320
}
312321
)
@@ -324,6 +333,8 @@ def applicable_review(report: dict[str, Any], correctness: dict[str, Any]) -> di
324333
entry is not None
325334
and entry.get("sources") == sources
326335
and entry.get("performance_impact") == "none"
336+
and entry.get("dependencies")
337+
== {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()}
327338
and entry.get("measurement_sha256") == measurement_digest(report)
328339
and entry.get("reviewed_correctness_sha256") == _json_sha256(correctness)
329340
):
@@ -353,6 +364,13 @@ def certification_shape_failures(provenance: dict[str, Any]) -> list[str]:
353364
for field in ("measurement_sha256", "reviewed_correctness_sha256"):
354365
if not _is_sha256(entry.get(field)):
355366
failures.append(f"Certification {index} lacks {field}")
367+
dependencies = entry.get("dependencies")
368+
if (
369+
not isinstance(dependencies, dict)
370+
or not dependencies
371+
or not all(_is_sha256(value) for value in dependencies.values())
372+
):
373+
failures.append(f"Certification {index} lacks dependency identities")
356374
for field in ("certified_at", "reason", "ml4t_commit"):
357375
value = entry.get(field)
358376
if not isinstance(value, str) or not value.strip():
@@ -413,6 +431,7 @@ def certify_source(
413431
"reason": reason.strip(),
414432
"sources": {name: _sha256(path) for name, path in TIMING_SOURCE_PATHS.items()},
415433
"performance_impact": "none",
434+
"dependencies": {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()},
416435
"measurement_sha256": measurement_digest(report),
417436
"reviewed_correctness_sha256": _json_sha256(correctness),
418437
}
@@ -501,6 +520,8 @@ def report_failures(report: dict[str, Any], correctness: dict[str, Any]) -> list
501520
provenance.get("ml4t_engine_source_sha256")
502521
== _tree_digest(PROJECT_ROOT / "src/ml4t/backtest")
503522
and provenance.get("sources") == expected_sources
523+
and provenance.get("dependencies")
524+
== {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()}
504525
)
505526
if not certification_failures and not measured_source and review is None:
506527
failures.append(

0 commit comments

Comments
 (0)