Skip to content

feat(moments): add sample_weight support to EmpiricalCovariance - #417

Merged
HugoDelatte merged 5 commits into
skfolio:mainfrom
nicoloangileri:feat/empirical-covariance-sample-weight
Oct 24, 2026
Merged

HugoDelatte merged 5 commits into
skfolio:mainfrom
nicoloangileri:feat/empirical-covariance-sample-weight

Conversation

@nicoloangileri

Copy link
Copy Markdown
Contributor

Reference Issues/PRs

Fixes #416. Builds on the sample-weight helpers from #412.

What does this implement/fix? Explain your changes.

Adds sample_weight to EmpiricalCovariance.fit, following the points in #416:

  • Validation and scale invariance: weights go through _validate_sample_weight and are rescaled with _normalize_sample_weight, so only relative sizes matter. Zero weights drop observations.
  • Location and assume_centered: the location is the weighted mean, or zero when assume_centered=True.
  • window_size: weights are validated against the full X, then sliced with the same apply_window_size call, so they stay aligned with the last window_size observations.
  • ddof: the weighted second moment is divided by 1 - ddof * sum(w**2) (weights summing to one). For ddof=0 this is the distributional covariance; for ddof=1 it is the reliability-weights correction already used by _weighted_variance and by np.cov(..., aweights=...). For uniform weights it reduces to n - ddof and matches the existing unweighted results.
  • Errors: all-zero weights, and weights whose effective number of observations 1 / sum(w**2) is not greater than ddof, raise a ValueError with an explicit message.
  • Routing: since fit now takes sample_weight, it can be requested with set_fit_request(sample_weight=True); a test checks it through CovarianceDistance and HierarchicalRiskParity.
  • Docs: the docstring explains ddof with weights and adds a Notes section on distributional versus sample-corrected covariance, saying that the correction does not by itself justify an unbiasedness claim for arbitrary scenario probabilities.

Default path: when sample_weight is None the code path is unchanged; the only addition is two is not None checks. Fit time on main vs this branch (best of 7 × 50 fits): 0.908 vs 0.905 ms for 2000×100, 0.103 vs 0.103 ms for 250×20.

Does your contribution introduce a new dependency? If yes, which one?

No.

What should a reviewer concentrate their feedback on?

The ddof convention with weights and the wording of the Notes section.

Any other comments?

Tests cover: uniform weights matching the unweighted estimator (ddof 0/1/3, assume_centered, window_size), agreement with np.cov(aweights=...), assume_centered, scale invariance, integer weights equal to repeated rows, zero weights, window slicing, a single asset, invalid weights, and routing through CovarianceDistance and HierarchicalRiskParity. All 26 new tests fail on main.

Locally: ruff check, ruff format, ty check and deptry src pass. The full suite passes except 34 tests (33 SCIP-based tests in test_mean_risk.py and the OpinionPooling doctest) that fail identically on main in my environment.

PR checklist

For all contributions

@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.90%. Comparing base (1213b8d) to head (a0df35e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #417   +/-   ##
=======================================
  Coverage   99.90%   99.90%           
=======================================
  Files         262      262           
  Lines       20751    20770   +19     
=======================================
+ Hits        20732    20751   +19     
  Misses         19       19           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nicoloangileri

Copy link
Copy Markdown
Contributor Author

Hi @HugoDelatte, this is ready for review when you have time: sample_weight support in EmpiricalCovariance, as discussed in #416, with CI green. #384 and #398 are also still open if you get a chance. Thanks!

@lcrmorin

Copy link
Copy Markdown
Contributor

Looks good overall.

One minor issue if that matter. There are some edge case that can produce near 0 denominator and can make the value explode.

With 25 observations, weights [1] * 24 + [0] and ddof=24, the denominator should be zero and trigger the existing ValueError. Rounding instead gives about 2.22e-16, which passes the check and produces a huge covariance.

This is unlikely in normal use, but a regression test and a guard against this boundary case could be useful.

With 24 equal weights and a zero weight, ddof=24 makes the correction
1 - ddof * sum(w**2) exactly zero, but rounding leaves eps, which passed
the check and gave a huge covariance. Treat a correction within
n_observations * eps of zero as zero.
@nicoloangileri

Copy link
Copy Markdown
Contributor Author

Thanks @lcrmorin, good catch. Fixed in 31c12bb: a correction 1 - ddof * sum(w**2) within n_observations * eps of zero is now treated as zero, so [1] * 24 + [0] with ddof=24 raises the existing ValueError. Your case is now a regression test, together with ddof=23 on the same weights, which still matches np.cov on the 24 weighted rows.

@HugoDelatte HugoDelatte self-assigned this Oct 23, 2026

@HugoDelatte HugoDelatte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.

It looked good, I've just simplified the weighted fit path, used a symmetric matrix product, and shortened the docs. Routing now defaults to WARN to preserve existing results until users explicitly opt in. Both validation checks remain, with added coverage for windowing and routing through pipelines, priors, HRP and Schur.

@HugoDelatte
HugoDelatte merged commit ef0cbfd into skfolio:main Oct 24, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ENH] Add sample-weight support to EmpiricalCovariance

3 participants