Repository navigation
feat(moments): add sample_weight support to EmpiricalCovariance - #417
HugoDelatte merged 5 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #417 +/- ##
=======================================
Coverage 99.90% 99.90%
=======================================
Files 262 262
Lines 20751 20770 +19
=======================================
+ Hits 20732 20751 +19
Misses 19 19 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Hi @HugoDelatte, this is ready for review when you have time: |
|
Looks good overall. One minor issue if that matter. There are some edge case that can produce near 0 denominator and can make the value explode. With 25 observations, weights This is unlikely in normal use, but a regression test and a guard against this boundary case could be useful. |
With 24 equal weights and a zero weight, ddof=24 makes the correction 1 - ddof * sum(w**2) exactly zero, but rounding leaves eps, which passed the check and gave a huge covariance. Treat a correction within n_observations * eps of zero as zero.
|
Thanks @lcrmorin, good catch. Fixed in 31c12bb: a correction |
HugoDelatte
left a comment
There was a problem hiding this comment.
Thanks for the PR.
It looked good, I've just simplified the weighted fit path, used a symmetric matrix product, and shortened the docs. Routing now defaults to WARN to preserve existing results until users explicitly opt in. Both validation checks remain, with added coverage for windowing and routing through pipelines, priors, HRP and Schur.
Reference Issues/PRs
Fixes #416. Builds on the sample-weight helpers from #412.
What does this implement/fix? Explain your changes.
Adds
sample_weighttoEmpiricalCovariance.fit, following the points in #416:_validate_sample_weightand are rescaled with_normalize_sample_weight, so only relative sizes matter. Zero weights drop observations.assume_centered: the location is the weighted mean, or zero whenassume_centered=True.window_size: weights are validated against the fullX, then sliced with the sameapply_window_sizecall, so they stay aligned with the lastwindow_sizeobservations.ddof: the weighted second moment is divided by1 - ddof * sum(w**2)(weights summing to one). Forddof=0this is the distributional covariance; forddof=1it is the reliability-weights correction already used by_weighted_varianceand bynp.cov(..., aweights=...). For uniform weights it reduces ton - ddofand matches the existing unweighted results.1 / sum(w**2)is not greater thanddof, raise aValueErrorwith an explicit message.fitnow takessample_weight, it can be requested withset_fit_request(sample_weight=True); a test checks it throughCovarianceDistanceandHierarchicalRiskParity.ddofwith weights and adds a Notes section on distributional versus sample-corrected covariance, saying that the correction does not by itself justify an unbiasedness claim for arbitrary scenario probabilities.Default path: when
sample_weightis None the code path is unchanged; the only addition is twois not Nonechecks. Fit time on main vs this branch (best of 7 × 50 fits): 0.908 vs 0.905 ms for 2000×100, 0.103 vs 0.103 ms for 250×20.Does your contribution introduce a new dependency? If yes, which one?
No.
What should a reviewer concentrate their feedback on?
The
ddofconvention with weights and the wording of the Notes section.Any other comments?
Tests cover: uniform weights matching the unweighted estimator (
ddof0/1/3,assume_centered,window_size), agreement withnp.cov(aweights=...),assume_centered, scale invariance, integer weights equal to repeated rows, zero weights, window slicing, a single asset, invalid weights, and routing throughCovarianceDistanceandHierarchicalRiskParity. All 26 new tests fail onmain.Locally:
ruff check,ruff format,ty checkanddeptry srcpass. The full suite passes except 34 tests (33 SCIP-based tests intest_mean_risk.pyand theOpinionPoolingdoctest) that fail identically onmainin my environment.PR checklist
For all contributions
fix(portfolio): ..., with a type listed in CONTRIBUTING.md.