Skip to content

Fix reloo k threshold - #399

Merged
jordandeklerk merged 3 commits into
arviz-devs:mainfrom
sheikhayaan:fix-reloo-k-threshold
Aug 4, 2026
Merged

jordandeklerk merged 3 commits into
arviz-devs:mainfrom
sheikhayaan:fix-reloo-k-threshold

Conversation

@sheikhayaan

Copy link
Copy Markdown
Contributor

Fixes #398

What changed:
Changed the default value of k_threshold from -np.inf to None in reloo.py to match the documentation, as requested in the issue.

@read-the-docs-community

read-the-docs-community Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

@jordandeklerk jordandeklerk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! We need to remove the irrelevant changes for the HDI error from this PR.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes are not relevant to this PR. Remove these.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Thanks for the review! I've removed the unrelated array.py changes and updated the docstring with your suggested text. Please take another look when you get a chance."

Comment thread src/arviz_stats/loo/reloo.py Outdated
Comment on lines 101 to 104

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that since we're changing the default behavior of reloo, it might be a good idea to be mroe explicit here about how to enable automatic refitting for all values.

Suggested change
As an extreme case, setting ``k_threshold=-np.inf`` will flag every observation and
make ``reloo`` perform the whole exact LOO-CV, requiring one refit per observation.
This is not generally recommended nor intended, however, if needed, this function can
be used to achieve the result.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Done — updated the docstring with your suggested text."

@codecov-commenter

codecov-commenter commented Aug 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.84%. Comparing base (ef043f4) to head (2ffe96b).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #399   +/-   ##
=======================================
  Coverage   82.84%   82.84%           
=======================================
  Files          44       44           
  Lines        6524     6524           
=======================================
  Hits         5405     5405           
  Misses       1119     1119           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jordandeklerk
jordandeklerk merged commit aa59e48 into arviz-devs:main Aug 4, 2026
11 checks passed
TeemuSailynoja pushed a commit to TeemuSailynoja/arviz-stats that referenced this pull request Aug 29, 2026
* fix: add helpful error message when hdi_prob is used instead of prob

* Fix arviz-devs#398: Set default k_threshold to None in reloo

* fix: remove unrelated hdi changes and update reloo docstring
aloctavodia added a commit that referenced this pull request Sep 10, 2026
* Add LFO-CV implementation with PSIS approximation

Implements Leave-Future-Out Cross-Validation for time series models using
forward-mode validation with Pareto-Smoothed Importance Sampling (PSIS)
approximation to reduce computational cost. Supports both exact (always refit)
and approximate methods.

- lfo_cv.py: Main LFO-CV function with approx/exact methods
- helper_lfo_cv.py: Helper functions for input validation, step computation,
  and result combination

* Integrate LFO-CV into package exports and compare function

Exports lfo_cv at package level and adds LFO-CV support to compare()
function with proper validation to prevent comparing LFO-CV with
incompatible methods (LOO-CV, K-fold-CV).

- __init__.py files: Export lfo_cv function
- compare.py: Add LFO-CV validation in _calculate_ics

* Add comprehensive LFO-CV test suite

Adds mock SamplingWrapper fixtures for LFO-CV testing along with
comprehensive unit tests for main function and helper utilities.

- conftest.py: LFO-CV fixtures (data_with_time, mock_lfo_wrapper, etc.)
- test_lfo_cv.py: Unit tests for lfo_cv function
- test_helper_lfo_cv.py: Unit tests for helper functions

* REF: fix the compare logic for comparing lfo-cv

* REF: reorient LFO-CV functionality and tests

* REF: update LFO-CV helper function to use negated log ratios for PSIS weights

* REF: reorganize and improve docs

* TEST: add a few more tests for LFO-CV

* REF: fix LFO-CV comparison logic and add validation for mixed settings

* REF: enhance LFO-CV validation and improve error messages

* DOC: clarify refit behavior

* DOC: add more clarity to the forecast horizon note

* DOC: better main docstring

* REF: fix alphabetical order

* Fix `bayesian_r2` silently using Bernoulli pseudo-variance when scale is omitted. (#389)

* adjust_rsquare_default_scale_warning

* docs: allow expected warning in bayesian_r2 Bernoulli example

* Update tests/test_metrics.py

Co-authored-by: Jordan Deklerk <[email protected]>

* Update tests/test_metrics.py

Co-authored-by: Jordan Deklerk <[email protected]>

* Update src/arviz_stats/base/diagnostics.py

Co-authored-by: Jordan Deklerk <[email protected]>

* Update src/arviz_stats/base/diagnostics.py

Co-authored-by: Jordan Deklerk <[email protected]>

* decorate test_bayesian_r2_continuous_no_scale_errors with ignore::UserWarning

---------

Co-authored-by: Jordan Deklerk <[email protected]>

* Make pred_mean and scale required in bayesian_r2 (#391)

* update coo_matrix to coo_array (#394)

* Build(deps): bump actions/checkout in the actions group (#396)

Bumps the actions group with 1 update: [actions/checkout](https://github.com/actions/checkout).


Updates `actions/checkout` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@9c091bb...3d3c42e)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Build(deps): bump https://github.com/astral-sh/ruff-pre-commit (#397)

Bumps the pre-commit group with 1 update: [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit).


Updates `https://github.com/astral-sh/ruff-pre-commit` from v0.15.21 to 0.15.22
- [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases)
- [Commits](astral-sh/ruff-pre-commit@01a675e...2700fd5)

---
updated-dependencies:
- dependency-name: https://github.com/astral-sh/ruff-pre-commit
  dependency-version: 0.15.22
  dependency-type: direct:production
  dependency-group: pre-commit
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix reloo k threshold (#399)

* fix: add helpful error message when hdi_prob is used instead of prob

* Fix #398: Set default k_threshold to None in reloo

* fix: remove unrelated hdi changes and update reloo docstring

* fix: convert ary to mutable numpy array in _ps_tail to support JAX arrays (#368)

* fix: convert ary to mutable numpy array at top of _ps_tail

JAX arrays are immutable and do not support in-place item assignment
(`ary[idx] = val` raises TypeError). When JAX is installed, xarray's
apply_ufunc can dispatch array operations through JAX's numpy backend,
causing `ary` to be a JAX array by the time it reaches `_ps_tail`.

Adding `ary = np.asarray(ary).copy()` at the entry of `_ps_tail`
ensures it always operates on a mutable numpy array. A regression test
is included that passes a JAX array through `loo()` directly.

* Add jax to the full tox env

The full env sets ARVIZ_REQUIRE_ALL_DEPS=TRUE so importorskip raises
instead of skipping, but jax was never installed there, so
test_loo_jax_arrays has been failing CI with ModuleNotFoundError.

* test: sample sequentially in mm_from_pymc log-likelihood test

With jax installed in the full CI env, jax registers an at-fork handler
that emits a RuntimeWarning when pm.sample forks worker processes, and
filterwarnings = ["error"] turns that into a test failure. Passing
cores=1 keeps sampling in-process, so no fork happens.

* Add moment matching arg to main `loo()` function (#386)

* FEAT: add moment matching to main loo function

* REF: update tests

* REF: simplify loo function parameters

* FIX: pass contiguous per-variable inputs to compiled moment matching functions

* DOC: fix wording of docstring

* REF: replace ESS calculation with azstats method (#401)

* fix: add helpful error message when hdi_prob is passed instead of prob (#392)

* fix: add helpful error message when hdi_prob is used instead of prob

* fix: update hdi_prob error to link directly to migration guide section

* test: add test for helpful hdi_prob error message

* test: fix hdi_prob test to use correct import and match pattern

* Prepare release (#402)

* prepare release

* prepare release

* update changelog for v1.3.0 changes (#403)

This commit was created by changelog-from-release in 'Post-release' CI workflow

Co-authored-by: aloctavodia <[email protected]>

* clean post relase (#404)

* Fix NaN in `loo_score` CRPS and simplify score calculations (#405)

* REF: optimize LOO score calculations and improve docs

* REF: update LOO score docs and tests

* REF: refactor LOO score tests

* Fix truncated Cauchy combination: restore correct denominator (#409) (#410)

* Fix truncated Cauchy combination: restore Eq. 24 denominator (#409)

The refactor in #352 changed the truncated Cauchy combination from a
masked mean (truncated sum divided by the total number of terms, as in
Eq. 24 of arXiv:2603.02928) to a mean over the retained subset, roughly
doubling the statistic and inflating the rejection rate of pot_c and
prit_c under the null. It also made the no-retained-terms case (every
pointwise p >= 0.5) undefined, guarded by a ValueError that a single
too-uniform column could trigger, aborting whole batched calls.

Restore the Eq. 24 semantics via np.where (avoiding the inf * 0 hazard
of the original mask multiplication), so the empty-truncation case is
simply a statistic of 0 (p = 0.5) and the error case disappears. Update
the test helper and expectations accordingly, and add regression
guards: grid input returns 0.5 instead of raising, one such column no
longer aborts a batched call, and the attained size under the null
stays below the regressed level.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Update tests/test_uniformity.py

* Update tests/test_uniformity.py

* Apply suggestion from @aloctavodia

* Apply suggestion from @aloctavodia

---------

Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: Osvaldo A Martin <[email protected]>

* prepare patch release (#412)

* Update changelog for v1.3.1 (#413)

* update changelog for v1.3.1 changes

    This commit was created by changelog-from-release in 'Post-release' CI workflow

* update versions and fix post-relese duplicated trigger

---------

Co-authored-by: OriolAbril <[email protected]>
Co-authored-by: Oriol (VANT Edge) <[email protected]>

* Addressing review comments

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Jordan Deklerk <[email protected]>
Co-authored-by: CamSalda <[email protected]>
Co-authored-by: Jordan Deklerk <[email protected]>
Co-authored-by: Oriol Abril-Pla <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: sheikhayaan <[email protected]>
Co-authored-by: Raphael Erik Hviding <[email protected]>
Co-authored-by: Osvaldo A Martin <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: aloctavodia <[email protected]>
Co-authored-by: Jorge Rotter <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
Co-authored-by: OriolAbril <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default value of k_threshold in reloo is inconsistent with the documentation

3 participants