Skip to content

Commit 7233675

Browse files
committed
fix(validation): compare locked dependencies across prerelease qualification
Reproduce the shared Python 3.15 failure through uv resolution. Bind selected packages, artifacts, markers and lock declarations while excluding the resolver prerelease permission alone. Preserve raw identities and all 17 accepted measurement records, and keep actual dependency drift rejection. Full suite: 2320 passed, 14 skipped. Fixes qualification of #151.
1 parent 1919bd8 commit 7233675

9 files changed

Lines changed: 139 additions & 10 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -360,7 +360,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
360360

361361
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
362362

363-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
363+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
364364

365365
### Synthetic diagnostic scenarios
366366

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
179179

180180
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
181181

182-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
182+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
183183

184184
### Synthetic diagnostic scenarios
185185

‎docs/user-guide/profiles.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
218218

219219
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
220220

221-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
221+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
222222

223223
### Synthetic diagnostic scenarios
224224

‎tests/contracts/test_real_strategy_runner.py‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -642,3 +642,53 @@ def measured_side(*, case_study, framework, side, **_settings):
642642
"performance-impact review" in failure
643643
for failure in benchmark.report_failures(report, correctness)
644644
)
645+
646+
647+
def test_qualification_resolver_permission_preserves_unchanged_measured_dependencies(
648+
tmp_path, monkeypatch
649+
) -> None:
650+
"""uv's actual prerelease resolve must not invalidate an unchanged package graph."""
651+
import os
652+
import shutil
653+
import tomllib
654+
655+
benchmark = _load_benchmark()
656+
validation = Path(__file__).parents[2] / "validation"
657+
project = validation.parent
658+
for name in ("pyproject.toml", "uv.lock", "README.md"):
659+
shutil.copy2(project / name, tmp_path / name)
660+
shutil.copytree(project / "src", tmp_path / "src", ignore=shutil.ignore_patterns("__pycache__"))
661+
dependency = "uv.lock"
662+
monkeypatch.setitem(benchmark.DEPENDENCY_PATHS, dependency, tmp_path / "uv.lock")
663+
correctness = json.loads((validation / "REAL_STRATEGY_RESULTS.json").read_text())
664+
report = json.loads((validation / "REAL_STRATEGY_PERFORMANCE.json").read_text())
665+
benchmark.certify_source(
666+
report,
667+
correctness,
668+
validation / "REAL_STRATEGY_RESULTS.json",
669+
"Review unchanged selected packages through shared qualification's resolver permission.",
670+
performance_impact="none",
671+
)
672+
assert benchmark.report_failures(report, correctness) == []
673+
before = tomllib.loads((tmp_path / "uv.lock").read_text())
674+
before_bytes = (tmp_path / "uv.lock").read_bytes()
675+
completed = subprocess.run(
676+
["uv", "lock", "--directory", str(tmp_path), "--prerelease", "allow", "--offline"],
677+
env={**os.environ, "SETUPTOOLS_SCM_PRETEND_VERSION": "0.1.13"},
678+
check=False,
679+
capture_output=True,
680+
text=True,
681+
)
682+
assert completed.returncode == 0, completed.stderr
683+
after = tomllib.loads((tmp_path / "uv.lock").read_text())
684+
assert (tmp_path / "uv.lock").read_bytes() != before_bytes
685+
assert after.pop("options") == {"prerelease-mode": "allow"}
686+
assert after == before # No selected version, artifact, hash, marker or source changed.
687+
assert benchmark.report_failures(report, correctness) == []
688+
689+
# Another real uv lock selects materially different dependencies and must fail.
690+
shutil.copy2(validation / "environments/vectorbt_pro_2025_12_31/uv.lock", tmp_path / "uv.lock")
691+
assert any(
692+
"performance-impact review" in failure
693+
for failure in benchmark.report_failures(report, correctness)
694+
)

‎validation/METHODOLOGY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -400,7 +400,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
400400

401401
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
402402

403-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
403+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
404404

405405
### Synthetic diagnostic scenarios
406406

‎validation/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
7676

7777
Measured 2026-10-13 on `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
7878

79-
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun.
79+
A performance-impact review retains these measurements for engine source `656210f6fe91`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
8080

8181
### Synthetic diagnostic scenarios
8282

‎validation/REAL_STRATEGY_PERFORMANCE.json‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,48 @@
155155
"vectorbt_pro": "c1817727bbf49d1551a072717db324c3e70484eefbdc27aa84af2a093e3e4d52",
156156
"zipline": "c23b547767fad7a60f8bd6b0abd1d8f1c1acf99ef0faab3da8fd34621b0ec482"
157157
}
158+
},
159+
{
160+
"certified_at": "2026-10-20T20:44:51.592393+00:00",
161+
"correctness_evidence_generated_at": "2026-10-17T01:17:00.495296+00:00",
162+
"correctness_evidence_sha256": "ba7f4978cdf675e3d7ff2c998a7eb2f13bc099cb86667d70d56879affdf46a3f",
163+
"correctness_pairs_passed": 17,
164+
"dependencies": {
165+
"uv.lock": "08ffb5f56cf383d8129333aeb0b784cd6f641b44b9afee678a139570a413a344",
166+
"validation/environments/backtrader/uv.lock": "b6716d65024b014103198b1feaec0706009ab4b70ad0ee42f185a576090cd6ae",
167+
"validation/environments/lean/uv.lock": "0af1e08ca7a021310f5afa34aaa724e38b0538ea4c9388f41ae1e2130de0fbc3",
168+
"validation/environments/vectorbt_oss/uv.lock": "cdd7f1882cbf3d283eebafbce4f93f1fcbf4a5eda6f938157e5f141e875c2961",
169+
"validation/environments/vectorbt_pro/uv.lock": "b7ebebc44a571d583ede63f27552594bd7c1b605f5faa9908df30da13be59c38",
170+
"validation/environments/vectorbt_pro_2025_12_31/uv.lock": "e8fed15af7d562ea60b8a320836c6199e098e5392bb4db622e68c8b8d8b6a0a1",
171+
"validation/environments/zipline/uv.lock": "43258cbcc10cdbc29c02da2d41603467b9386ee00f6ea73e501bbffad89d4de0"
172+
},
173+
"dependency_graphs": {
174+
"uv.lock": "ed7c38a3f7ab567bd44db7e1558328728fd24f958a6be9897f15f2cb848e7fb9",
175+
"validation/environments/backtrader/uv.lock": "6c9b64183ed42e58b91d753bd6eeaaa5366374db494ec66bad10bb5fa9ba7154",
176+
"validation/environments/lean/uv.lock": "f64352a45b275049d72e098df7deeea3c275d5eaf81ff1da1de84b66f9aabb95",
177+
"validation/environments/vectorbt_oss/uv.lock": "ceabca20fb1b89ccd70e3778bcec19fc32b1ccd52704fe6725d8f22b8280afb8",
178+
"validation/environments/vectorbt_pro/uv.lock": "0f0f61ac16d9bafc9c15394fd6c068d690dfa3d9bf25a80ba2d1f75df11db0ce",
179+
"validation/environments/vectorbt_pro_2025_12_31/uv.lock": "ed18504748215fcd7b16ca4ed2af45b16d8507654d9c809b5739eef24f645760",
180+
"validation/environments/zipline/uv.lock": "e0e84d5ee04be1fb25a7176933e3b20b5944aafd35ee84ed6b84163bae7ced2f"
181+
},
182+
"engine_source_sha256": "656210f6fe91eb6bac893598909f0363c5951e7bc8f7b14e15d8fb653fb23649",
183+
"measurement_sha256": "4bbc924aeeed357ab44a196c003ba7647c0d0faf60aee25ed534f7481ade1852",
184+
"ml4t_commit": "df10938aa2ae6ed54174d4a065860db196b7c647",
185+
"performance_impact": "none",
186+
"reason": "The only timing-boundary implementation change is dependency identity validation: shared qualification adds a prerelease resolver permission without changing any locked package, artifact, marker or environment. Runtime Engine, input preparation and comparison runners are unchanged. Retain original raw dependency hashes and bind the parsed lock graph; all actual dependency changes remain refused.",
187+
"reviewed_correctness_sha256": "cbeac90d998cd3aa7830ea4e3e715e03a6437452695d1d7fd99b79f4a8cfb5c2",
188+
"sources": {
189+
"backtrader": "a456cc3bf9ed6708d239f21ed2c2495a3c19a9a8306634fee24619c2c0484459",
190+
"benchmark": "28d04f313753a900e97e51cda70458a78ecfd9665f062df8aca6051e0b489746",
191+
"comparison_input": "a00bf9383c3e5561333ef0fa734745296cf9cda6fe7155ce950713c1ba2da4f9",
192+
"lean_crypto": "82f1fbb6054fba25d74a4eabbc9ad4eae808caf89c00f560edefa63304da0dc4",
193+
"lean_equity": "f566341d0f4103eda45280684be1b8a91f9da1a46a3a06477cba3b49674d89e8",
194+
"lean_fx": "85c18eb1368bfac938fd9909eaa481de867cd864b1f6852281da8af7c2b44951",
195+
"ml4t": "c8a5eb818a64a128dce41bc9827adea0a46fdab87a11f418abb19d5df8c7bff9",
196+
"vectorbt_oss": "c1817727bbf49d1551a072717db324c3e70484eefbdc27aa84af2a093e3e4d52",
197+
"vectorbt_pro": "c1817727bbf49d1551a072717db324c3e70484eefbdc27aa84af2a093e3e4d52",
198+
"zipline": "c23b547767fad7a60f8bd6b0abd1d8f1c1acf99ef0faab3da8fd34621b0ec482"
199+
}
158200
}
159201
],
160202
"ml4t_engine_source_sha256": "1e2bf2f45a835f8b0a81b59141843dae77b6341233d7c9ba494efada0a328a25",

‎validation/generate_parity_claims.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -236,7 +236,9 @@ def render_claims(
236236
f"`{review['engine_source_sha256'][:12]}`. The measured source, locked dependencies, "
237237
"review rationale and current correctness evidence are recorded with the raw samples. "
238238
"Measurements are repeated when changes materially affect timed execution; "
239-
"a new source revision alone does not require a benchmark rerun.",
239+
"a new source revision alone does not require a benchmark rerun. "
240+
"Dependency validation checks selected packages and artifacts; allowing prereleases "
241+
"without changing those selections does not invalidate the measurements.",
240242
]
241243
if review is not None
242244
else []

‎validation/real_strategy_benchmark.py‎

Lines changed: 39 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -275,6 +275,7 @@ def build_report(
275275
"ml4t_engine_source_sha256": _tree_digest(PROJECT_ROOT / "src/ml4t/backtest"),
276276
"sources": {name: _sha256(path) for name, path in TIMING_SOURCE_PATHS.items()},
277277
"dependencies": {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()},
278+
"dependency_graphs": dependency_graphs(),
278279
},
279280
"correctness_evidence_generated_at": correctness["generated_at"],
280281
"records": records,
@@ -307,6 +308,33 @@ def _json_sha256(payload: object) -> str:
307308
).hexdigest()
308309

309310

311+
def dependency_graphs() -> dict[str, str]:
312+
"""Bind locked packages/artifacts while allowing an unchanged prerelease resolution.
313+
314+
Shared qualification resolves with ``--prerelease allow``. uv records that
315+
resolver permission even when every selected package and artifact is unchanged.
316+
All other lock declarations, markers, package records and hashes remain bound.
317+
Raw lock hashes are retained separately as the original provenance.
318+
"""
319+
graphs = {}
320+
for name, path in DEPENDENCY_PATHS.items():
321+
lock = tomllib.loads(path.read_text(encoding="utf-8"))
322+
options = lock.get("options", {})
323+
options.pop("prerelease-mode", None)
324+
if not options:
325+
lock.pop("options", None)
326+
graphs[name] = _json_sha256(lock)
327+
return graphs
328+
329+
330+
def dependencies_match(provenance: dict[str, Any]) -> bool:
331+
if "dependency_graphs" in provenance:
332+
return provenance["dependency_graphs"] == dependency_graphs()
333+
return provenance.get("dependencies") == {
334+
name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()
335+
}
336+
337+
310338
def measurement_digest(report: dict[str, Any]) -> str:
311339
"""Identify the original measurements without including subsequent reviews."""
312340
return _json_sha256(
@@ -333,8 +361,7 @@ def applicable_review(report: dict[str, Any], correctness: dict[str, Any]) -> di
333361
entry is not None
334362
and entry.get("sources") == sources
335363
and entry.get("performance_impact") == "none"
336-
and entry.get("dependencies")
337-
== {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()}
364+
and dependencies_match(entry)
338365
and entry.get("measurement_sha256") == measurement_digest(report)
339366
and entry.get("reviewed_correctness_sha256") == _json_sha256(correctness)
340367
):
@@ -371,6 +398,14 @@ def certification_shape_failures(provenance: dict[str, Any]) -> list[str]:
371398
or not all(_is_sha256(value) for value in dependencies.values())
372399
):
373400
failures.append(f"Certification {index} lacks dependency identities")
401+
graphs = entry.get("dependency_graphs")
402+
if graphs is not None and (
403+
not isinstance(graphs, dict)
404+
or not isinstance(dependencies, dict)
405+
or graphs.keys() != dependencies.keys()
406+
or not all(_is_sha256(value) for value in graphs.values())
407+
):
408+
failures.append(f"Certification {index} has invalid dependency graph identities")
374409
for field in ("certified_at", "reason", "ml4t_commit"):
375410
value = entry.get(field)
376411
if not isinstance(value, str) or not value.strip():
@@ -432,6 +467,7 @@ def certify_source(
432467
"sources": {name: _sha256(path) for name, path in TIMING_SOURCE_PATHS.items()},
433468
"performance_impact": "none",
434469
"dependencies": {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()},
470+
"dependency_graphs": dependency_graphs(),
435471
"measurement_sha256": measurement_digest(report),
436472
"reviewed_correctness_sha256": _json_sha256(correctness),
437473
}
@@ -520,8 +556,7 @@ def report_failures(report: dict[str, Any], correctness: dict[str, Any]) -> list
520556
provenance.get("ml4t_engine_source_sha256")
521557
== _tree_digest(PROJECT_ROOT / "src/ml4t/backtest")
522558
and provenance.get("sources") == expected_sources
523-
and provenance.get("dependencies")
524-
== {name: _sha256(path) for name, path in DEPENDENCY_PATHS.items()}
559+
and dependencies_match(provenance)
525560
)
526561
if not certification_failures and not measured_source and review is None:
527562
failures.append(

0 commit comments

Comments
 (0)