Repository navigation
Fix reloo k threshold - #399
Conversation
Documentation build overview
13 files changed ·
|
jordandeklerk
left a comment
There was a problem hiding this comment.
Thanks for this! We need to remove the irrelevant changes for the HDI error from this PR.
There was a problem hiding this comment.
These changes are not relevant to this PR. Remove these.
There was a problem hiding this comment.
"Thanks for the review! I've removed the unrelated array.py changes and updated the docstring with your suggested text. Please take another look when you get a chance."
There was a problem hiding this comment.
Note that since we're changing the default behavior of reloo, it might be a good idea to be mroe explicit here about how to enable automatic refitting for all values.
| As an extreme case, setting ``k_threshold=-np.inf`` will flag every observation and | |
| make ``reloo`` perform the whole exact LOO-CV, requiring one refit per observation. | |
| This is not generally recommended nor intended, however, if needed, this function can | |
| be used to achieve the result. |
There was a problem hiding this comment.
"Done — updated the docstring with your suggested text."
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #399 +/- ##
=======================================
Coverage 82.84% 82.84%
=======================================
Files 44 44
Lines 6524 6524
=======================================
Hits 5405 5405
Misses 1119 1119 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
* fix: add helpful error message when hdi_prob is used instead of prob * Fix arviz-devs#398: Set default k_threshold to None in reloo * fix: remove unrelated hdi changes and update reloo docstring
* Add LFO-CV implementation with PSIS approximation Implements Leave-Future-Out Cross-Validation for time series models using forward-mode validation with Pareto-Smoothed Importance Sampling (PSIS) approximation to reduce computational cost. Supports both exact (always refit) and approximate methods. - lfo_cv.py: Main LFO-CV function with approx/exact methods - helper_lfo_cv.py: Helper functions for input validation, step computation, and result combination * Integrate LFO-CV into package exports and compare function Exports lfo_cv at package level and adds LFO-CV support to compare() function with proper validation to prevent comparing LFO-CV with incompatible methods (LOO-CV, K-fold-CV). - __init__.py files: Export lfo_cv function - compare.py: Add LFO-CV validation in _calculate_ics * Add comprehensive LFO-CV test suite Adds mock SamplingWrapper fixtures for LFO-CV testing along with comprehensive unit tests for main function and helper utilities. - conftest.py: LFO-CV fixtures (data_with_time, mock_lfo_wrapper, etc.) - test_lfo_cv.py: Unit tests for lfo_cv function - test_helper_lfo_cv.py: Unit tests for helper functions * REF: fix the compare logic for comparing lfo-cv * REF: reorient LFO-CV functionality and tests * REF: update LFO-CV helper function to use negated log ratios for PSIS weights * REF: reorganize and improve docs * TEST: add a few more tests for LFO-CV * REF: fix LFO-CV comparison logic and add validation for mixed settings * REF: enhance LFO-CV validation and improve error messages * DOC: clarify refit behavior * DOC: add more clarity to the forecast horizon note * DOC: better main docstring * REF: fix alphabetical order * Fix `bayesian_r2` silently using Bernoulli pseudo-variance when scale is omitted. (#389) * adjust_rsquare_default_scale_warning * docs: allow expected warning in bayesian_r2 Bernoulli example * Update tests/test_metrics.py Co-authored-by: Jordan Deklerk <[email protected]> * Update tests/test_metrics.py Co-authored-by: Jordan Deklerk <[email protected]> * Update src/arviz_stats/base/diagnostics.py Co-authored-by: Jordan Deklerk <[email protected]> * Update src/arviz_stats/base/diagnostics.py Co-authored-by: Jordan Deklerk <[email protected]> * decorate test_bayesian_r2_continuous_no_scale_errors with ignore::UserWarning --------- Co-authored-by: Jordan Deklerk <[email protected]> * Make pred_mean and scale required in bayesian_r2 (#391) * update coo_matrix to coo_array (#394) * Build(deps): bump actions/checkout in the actions group (#396) Bumps the actions group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@9c091bb...3d3c42e) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build(deps): bump https://github.com/astral-sh/ruff-pre-commit (#397) Bumps the pre-commit group with 1 update: [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit). Updates `https://github.com/astral-sh/ruff-pre-commit` from v0.15.21 to 0.15.22 - [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases) - [Commits](astral-sh/ruff-pre-commit@01a675e...2700fd5) --- updated-dependencies: - dependency-name: https://github.com/astral-sh/ruff-pre-commit dependency-version: 0.15.22 dependency-type: direct:production dependency-group: pre-commit ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix reloo k threshold (#399) * fix: add helpful error message when hdi_prob is used instead of prob * Fix #398: Set default k_threshold to None in reloo * fix: remove unrelated hdi changes and update reloo docstring * fix: convert ary to mutable numpy array in _ps_tail to support JAX arrays (#368) * fix: convert ary to mutable numpy array at top of _ps_tail JAX arrays are immutable and do not support in-place item assignment (`ary[idx] = val` raises TypeError). When JAX is installed, xarray's apply_ufunc can dispatch array operations through JAX's numpy backend, causing `ary` to be a JAX array by the time it reaches `_ps_tail`. Adding `ary = np.asarray(ary).copy()` at the entry of `_ps_tail` ensures it always operates on a mutable numpy array. A regression test is included that passes a JAX array through `loo()` directly. * Add jax to the full tox env The full env sets ARVIZ_REQUIRE_ALL_DEPS=TRUE so importorskip raises instead of skipping, but jax was never installed there, so test_loo_jax_arrays has been failing CI with ModuleNotFoundError. * test: sample sequentially in mm_from_pymc log-likelihood test With jax installed in the full CI env, jax registers an at-fork handler that emits a RuntimeWarning when pm.sample forks worker processes, and filterwarnings = ["error"] turns that into a test failure. Passing cores=1 keeps sampling in-process, so no fork happens. * Add moment matching arg to main `loo()` function (#386) * FEAT: add moment matching to main loo function * REF: update tests * REF: simplify loo function parameters * FIX: pass contiguous per-variable inputs to compiled moment matching functions * DOC: fix wording of docstring * REF: replace ESS calculation with azstats method (#401) * fix: add helpful error message when hdi_prob is passed instead of prob (#392) * fix: add helpful error message when hdi_prob is used instead of prob * fix: update hdi_prob error to link directly to migration guide section * test: add test for helpful hdi_prob error message * test: fix hdi_prob test to use correct import and match pattern * Prepare release (#402) * prepare release * prepare release * update changelog for v1.3.0 changes (#403) This commit was created by changelog-from-release in 'Post-release' CI workflow Co-authored-by: aloctavodia <[email protected]> * clean post relase (#404) * Fix NaN in `loo_score` CRPS and simplify score calculations (#405) * REF: optimize LOO score calculations and improve docs * REF: update LOO score docs and tests * REF: refactor LOO score tests * Fix truncated Cauchy combination: restore correct denominator (#409) (#410) * Fix truncated Cauchy combination: restore Eq. 24 denominator (#409) The refactor in #352 changed the truncated Cauchy combination from a masked mean (truncated sum divided by the total number of terms, as in Eq. 24 of arXiv:2603.02928) to a mean over the retained subset, roughly doubling the statistic and inflating the rejection rate of pot_c and prit_c under the null. It also made the no-retained-terms case (every pointwise p >= 0.5) undefined, guarded by a ValueError that a single too-uniform column could trigger, aborting whole batched calls. Restore the Eq. 24 semantics via np.where (avoiding the inf * 0 hazard of the original mask multiplication), so the empty-truncation case is simply a statistic of 0 (p = 0.5) and the error case disappears. Update the test helper and expectations accordingly, and add regression guards: grid input returns 0.5 instead of raising, one such column no longer aborts a batched call, and the attained size under the null stays below the regressed level. Co-Authored-By: Claude Fable 5 <[email protected]> * Update tests/test_uniformity.py * Update tests/test_uniformity.py * Apply suggestion from @aloctavodia * Apply suggestion from @aloctavodia --------- Co-authored-by: Claude Fable 5 <[email protected]> Co-authored-by: Osvaldo A Martin <[email protected]> * prepare patch release (#412) * Update changelog for v1.3.1 (#413) * update changelog for v1.3.1 changes This commit was created by changelog-from-release in 'Post-release' CI workflow * update versions and fix post-relese duplicated trigger --------- Co-authored-by: OriolAbril <[email protected]> Co-authored-by: Oriol (VANT Edge) <[email protected]> * Addressing review comments --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Jordan Deklerk <[email protected]> Co-authored-by: CamSalda <[email protected]> Co-authored-by: Jordan Deklerk <[email protected]> Co-authored-by: Oriol Abril-Pla <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: sheikhayaan <[email protected]> Co-authored-by: Raphael Erik Hviding <[email protected]> Co-authored-by: Osvaldo A Martin <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: aloctavodia <[email protected]> Co-authored-by: Jorge Rotter <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]> Co-authored-by: OriolAbril <[email protected]>
Fixes #398
What changed:
Changed the default value of
k_thresholdfrom-np.inftoNoneinreloo.pyto match the documentation, as requested in the issue.