Skip to content

Overfitting controls: non-gating RiskLabAI 3.2.0 parity test, with upstream's native expected-max Sharpe and DSR as oracles - #729

Merged
seathatflowsinourveins merged 7 commits into
mainfrom
trading/overfitting-riskl-parity-20261005
Oct 5, 2026
Merged

Overfitting controls: non-gating RiskLabAI 3.2.0 parity test, with upstream's native expected-max Sharpe and DSR as oracles#729
seathatflowsinourveins merged 7 commits into
mainfrom
trading/overfitting-riskl-parity-20261005

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes: it adds a non-gating parity test of the overfitting controls (PSR, expected-max Sharpe/DSR, CSCV/PBO) against RiskLabAI 3.2.0. The test runs in a hash-locked research env, pins each known semantic difference at its upstream line, and records which capabilities upstream already ships natively, under the 2026-10-05 never-rebuild rule. overfitting.py, the frozen parameters, the evaluators and the historical receipt are unchanged.
  • Base commit: f946c6d4ca988a17b6fa4392ecb488909f147883
  • Lane: lane:trading
  • Owned paths touched:
    • tests/test_overfitting_riskl_parity.py
    • blueprints/us-equities/overfitting-controls/{README.md, riskl-parity-receipt.json, requirements-riskl-parity.in, requirements-riskl-parity.lock}
    • docs/decisions/2026-10-05-overfitting-riskl-parity.md
    • .github/osv-scanner-lockfiles.json, which carries the lock inventory entry
    • manifests/evidence.json (last commit, per the hot-file protocol)

SOTA sources

  • RiskLabAI/RiskLabAI.py 3.2.0 at 7d5aa11271c70d83a7069d5900d80bb08b29e481 (https://github.com/RiskLabAI/RiskLabAI.py):

    • src/RiskLabAI/backtest/probabilistic_sharpe_ratio.py:78-96 and :133-144
    • src/RiskLabAI/backtest/test_set_overfitting.py:12-56 (expected_max_sharpe_ratio, effective N)
    • src/RiskLabAI/validation/leakage_aware_hpo.py:177-240 (deflated_sharpe_gate)
    • src/RiskLabAI/backtest/probability_of_backtest_overfitting.py:69 (ordinal ranks) and the CSCV split

    Source sha256s are recorded in the receipt.

  • NumPy 2.5.3 argsort default kind (quicksort), for the tie-order expectation.

  • google/osv-scanner v2.6.0, from the checksum-verified release recipe in .github/workflows/security-scan.yml, for the lock's required OSV scan.

Evidence-class table

Claim Evidence class Command / receipt
Numeric parity in the regular case, plus each documented divergence local_integration (unchanged upstream library, locally authored test) python -m unittest tests.test_overfitting_riskl_parity in the hash-locked parity venv: 14 passed, 0 skips; riskl-parity-receipt.json
RiskLabAI's own selected tests pass in the parity env upstream tests (unchanged) 12 upstream PSR/PBO/expected-max/DSR tests passed; two Optuna HPO tests deselected
The test is non-gating in CI without RiskLabAI local_integration the same module in an env without RiskLabAI: OK (skipped=14)
Each divergence test fails when its difference is removed synthetic (scratch mutants) 7 isolated scratch mutants, each exit 1 as intended
The lock passes the required OSV scan local_integration osv-scanner 2.6.0: 19 packages, no issues

Local commands run

$ python3 -m unittest tests.test_overfitting_riskl_parity tests.test_overfitting_controls tests.test_osv_lockfile_coverage   Ran 85, OK (skipped=14)   [coordinator]
$ python3 scripts/validate.py                    exit 0   [coordinator]
$ python3 scripts/evidence_manifest.py --check   exit 0   [coordinator]
$ git diff --check                               exit 0   [coordinator]

Review

Built by GPT-6.1 Sol through the packaged SDK worker (rounds t-parity-r1 and t-parity-r2); committed by the coordinator.

  • r1 read. A Claude Opus cross-family read returned FINDINGS: 1 P1 and 4 P2. The P1 was that upstream's effective-N expected-max Sharpe and its native DSR had been missed.
  • r2 delta read. Round r2 fixed all five. The Opus delta read of 6b060b140..63531f18c returned ACCEPT: 0 P1, 2 P2. The P2s are recorded below as residuals.
  • Related record. The private job-073 adjudication received a dated correction note for the effective-N and DSR claim.

Decision record

docs/decisions/2026-10-05-overfitting-riskl-parity.md

Checklist

  • No GitHub Actions or workflow changes.
  • No secrets printed, logged or committed. No broker credentials, live endpoints or keys.
  • No new paid hosting, subscription or billing surface.
  • Peer-owned untracked files and worktrees were preserved.

Recorded residuals (bounded review, 2026-10-05)

Claude Opus delta read r2 at 63531f18c429 returned ACCEPT with no p1. Under the bounded review rule these p2 items land as recorded residuals, each owed as a follow-up:

  • docs/decisions/2026-10-05-overfitting-riskl-parity.md:119-139 (exclusions at :115-117); blueprints/us-equities/overfitting-controls/overfitt: The adoption record is incomplete under the 2026-10-05 never-rebuild rule.
  • tests/test_overfitting_riskl_parity.py:305-318 (the claims it supports are at blueprints/us-equities/overfitting-controls/riskl-parity-recei: Low severity.

🤖 Generated with Claude Code

Scout and others added 7 commits October 5, 2026 02:18
Scout @claude
Overfitting controls: non-gating parity test against RiskLabAI 3.2.0,…
76f1ce3 
… with the six known differences pinned

The repository keeps its own overfitting.py. This is glue under the 2026-10-05 never-rebuild
rule: the Claude adjudication of job 073 found that RiskLabAI 3.2.0 (RiskLabAI.py commit
7d5aa112) uses the same formulas in the regular case, but six semantic differences would
change gate results.

- tests/test_overfitting_riskl_parity.py is skipped unless RiskLabAI 3.2.0 is importable, so CI
  without it neither gates nor fails on it. In the regular case it checks numeric parity for
  PSR, the benchmark Sharpe/DSR and CSCV/PBO within a stated tolerance. It also has one test per
  known difference that asserts the documented divergence, so an upstream or local change fails
  loudly:
  1. a non-positive PSR denominator;
  2. N=1 and the effective number of trials;
  3. unequal np.array_split blocks;
  4. ordinal versus averaged ranks;
  5. strict PBO;
  6. std=0.
  Each difference is confirmed at the pinned upstream lines.
- A research-only parity env: requirements-riskl-parity.in and a hash-pinned lock on CPython
  3.12. RiskLabAI's own upstream tests ran in it, and a negative-control run shows the
  divergence tests fail when a difference is removed.
- The README and docs/decisions/2026-10-05-overfitting-riskl-parity.md record which side the
  repository keeps and why, and the trigger for adopting RiskLabAI directly: upstream support
  for an effective trial count and for strict PBO.

overfitting.py and the frozen protocols are unchanged.

Built by GPT-6.1 Sol through the packaged SDK worker (round t-parity-r1); committed by the
coordinator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Register the parity env's lock and .in in the osv-scanner lockfile in…
921dece 
…ventory

Coordinator follow-up: the pre-push lockfile-coverage test requires every tracked lockfile
and manifest to be listed. requirements-riskl-parity.lock (parser requirements.txt) and its
.in (covered by that lock) are added, and the inventory is re-registered.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Hot-file protocol: reset manifests/evidence.json to the merge base be…
7306cd1 
…fore the final hot-file commit

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Reapply this branch's manifests/evidence.json edits on the merge base…
6b060b1 
… (hot-file protocol: every hot-file edit in the last commit)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
RiskLabAI parity r2: compare against upstream expected-max Sharpe and…
ba3c4d1 
… DSR, tie-order-robust ranks, degenerate CSCV columns

- The parity test now compares against upstream expected_max_sharpe_ratio and deflated_sharpe_gate (RiskLabAI/RiskLabAI.py@7d5aa112, src/RiskLabAI/backtest/test_set_overfitting.py:12-56 and validation/leakage_aware_hpo.py:177-240).
  - It adds supplied-N parity for N in {1, 2, 5, 100}, plus native DSR comparisons on fixed series.
  - It records upstream's Euler-constant rounding, with relative tolerance 1e-11.
  - The capability claims, the receipt and the adoption rationale now say that independent-N expected-max Sharpe and DSR already exist natively.
- Tied ranks accept either valid upstream ordinal rank, 2 or 3, while requiring the local averaged rank of 2.5. A regression exercises the alternate tie order through the real upstream PBO.
- Constant and near-constant CSCV columns are exercised. The local code raises ValueError, while upstream returns a finite PBO. The local threshold std <= 1e-12 * max(1, |mean|) is documented, and an oracle-sensitivity regression covers it.
- The CSCV adoption trigger is narrowed to averaged ranks and the required diagnostics. The README states that the research lock takes part in the required OSV scan, with advisory response owned by the trading lane.

Frozen controls, parameters, evaluators, the historical receipt and the lock are unchanged.

Built by GPT-6.1 Sol through the packaged SDK worker (round t-parity-r2). Commands ran from a clean hash-locked scratch venv:
- 14 parity tests passed;
- 12 unchanged upstream tests passed;
- 7 scratch mutants each failed, as intended;
- osv-scanner 2.6.0 found no issues;
- validate, evidence_manifest --check and git diff --check exited 0.

Committed by the coordinator.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Hot-file protocol: reset manifests/evidence.json to the merge base be…
19f3ffe 
…fore the final hot-file commit

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Reapply this branch's manifests/evidence.json edits on the merge base…
63531f1 
… (hot-file protocol: every hot-file edit in the last commit)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@seathatflowsinourveins seathatflowsinourveins added the lane:trading Trading lane: sim, paper and live north star label Oct 5, 2026
@socket-security

socket-security Bot commented Oct 5, 2026

Copy link
Copy Markdown

Dependency limit exceeded — report not shown.

This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report.

Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard.

Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account.

@seathatflowsinourveins

seathatflowsinourveins commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Claude session native-agent-stack-5f: landing at head 63531f18c429753aa0323863edeb421d0eccc61f. The command center (wsl-architecture-design) gave its ACK at this exact head; under the user's 2026-10-04 decision it owns merges that touch the hot registry. The read-only cross-family review (GPT-6.1 Sol max, the packaged Codex SDK worker) returned FINDINGS at 6b060b140227; this head is reached from it through recorded carry edges (equal owned patch-ids, or a cross-family delta read returning ACCEPT at the edge's target), with its 1 P1 item(s) resolved in one repair round as recorded.

Observed main f3ebf469b49c3e8a41efd28d3d30d53b2edbbac1. coordination/merge_tree_landing_check.py (sha256 a11649fcf2e5844c…) exit 0:

main f3ebf469b49c3e8a41efd28d3d30d53b2edbbac1 head 63531f18c429753aa0323863edeb421d0eccc61f base f946c6d4ca988a17b6fa4392ecb488909f147883 merged-tree 7a3f6da2bf3ad616b74437d3256379265d93548c merge-tree-exit 0
ok   1: clean three-way merge
ok   2: merged-vs-main paths 8, outside PR-owned 0 []
ok   3: main drift 39 paths, overlap with PR-owned inputs (registry excluded) []
ok   4: registry foreign rows equal True, order preserved True, PR-owned rows 7, unowned top-level keys differing []
ok   5: merged files[] sorted by path with no duplicates: True (10104 rows)
LANDABLE
condition 5: merged files[] sorted and unique (tree 7a3f6da2bf3ad616b74437d3256379265d93548c)

Required checks at this head: 7 pass . Unresolved review threads: 0.

@seathatflowsinourveins
seathatflowsinourveins merged commit fead5d8 into main Oct 5, 2026
25 of 30 checks passed
@seathatflowsinourveins

seathatflowsinourveins commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Claude session native-agent-stack-5f: post-merge observation. Landed as fead5d8f8042f01ffc1a785f3b8ad6c410ccb506 (parent f3ebf469b49c3e8a41efd28d3d30d53b2edbbac1), tree 7a3f6da2bf3ad616b74437d3256379265d93548c. The landed tree equals the checked merged tree and the parent is the observed main.

@seathatflowsinourveins
seathatflowsinourveins deleted the trading/overfitting-riskl-parity-20261005 branch October 5, 2026 08:39
seathatflowsinourveins pushed a commit that referenced this pull request Oct 5, 2026
Scout @claude
Merge main cb33948 into the XNYS-primary branch (lockfile inventory m…
d18a3af 
…erged by entry; registry last)

Both sides added an entry at the same place in .github/osv-scanner-lockfiles.json (this branch's
adaptive-paper engine lock, main's #729 riskl-parity lock), so a line merge produced one object with two "path" keys
and the queue's line-union refresh stopped at record discovery, whose JSON reader refuses duplicate keys. The file is
merged by whole entries instead: main's lists plus this branch's three entries at their reviewed positions, so it
differs from main by exactly this branch's insertions and from the branch head by exactly main's.
.github/workflows/catalog-freshness.yml merged cleanly. manifests/evidence.json is main's registry with this branch's
rows re-registered from the merged files, plus its receipts and convergence-record additions (hot-file protocol).

Checks on the merged tree: scripts/validate.py exit 0; scripts/validate_convergence.py --all-recorded exit 0;
scripts/evidence_manifest.py --check passed (10,157 files); git diff --check against main clean. Unit tests: 410 run
with the system python3 (8 errors, all ModuleNotFoundError exchange_calendars); the 7 calendar-dependent modules
re-run in a venv from .github/requirements-calendar.txt: 591 tests OK (126 skipped).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

lane:trading Trading lane: sim, paper and live north star

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@seathatflowsinourveins