Skip to content

[ENH] Chen & Yang (2004) forecast accuracy metrics (#8681) - #9364

Merged
fkiraly merged 13 commits into
sktime:mainfrom
michaelellis003:feat/msmape-metric
Jul 26, 2026
Merged

fkiraly merged 13 commits into
sktime:mainfrom
michaelellis003:feat/msmape-metric

Conversation

@michaelellis003

Copy link
Copy Markdown
Contributor

Reference Issues/PRs

Fixes #8681

What does this implement/fix? Explain your changes.

Adds 7 forecast accuracy metrics from Section 4 of Chen & Yang (2004) "Assessing Forecast Accuracy Measures":

  • NormalizedMeanSquaredError (NMSE) — RMSE normalized by standard deviation of actuals
  • InterQuartileRangeError (IQR Error) — RMSE normalized by interquartile range of actuals
  • KLDivergenceNormal (KL-N) — KL-divergence assuming normal errors, with optional windowed variants (KL-N1 window=5, KL-N2 window=10)
  • KLDivergenceSingleExponential (KL-DE1) — KL-divergence assuming Laplace errors, rolling std scale
  • KLDivergenceDoubleExponential (KL-DE2) — KL-divergence assuming Laplace errors, rolling MAD scale
  • MeanAbsolutePercentageErrorStabilized (msMAPE) — symmetric MAPE stabilized by rolling MAD
  • TheilU2 — Theil's U2 statistic (forecast MSE relative to naive seasonal walk)

All metrics support multioutput, evaluate, and evaluate_by_index. Metrics with non-linear aggregation (NMSE, IQR, KL-N, TheilU2) use jackknife leave-one-out pseudo-values for evaluate_by_index. Linear-mean metrics (KL-DE1, KL-DE2, msMAPE) return per-index values directly.

Files added/changed:

  • _nmse.py, _iqre.py, _kln.py, _klde_base.py, _klde1.py, _klde2.py, _mapes.py, _theilu2.py
  • init.py — registered all 7 classes
  • docs/source/api_reference/performance_metrics.rst — added to API reference
  • sktime/tests/_config.py — added doctest exclusions for np.float64() formatting
  • tests/test_metrics.py — added unit tests for all 7 metrics + evaluate_by_index index correctness test

Does your contribution introduce a new dependency? If yes, which one?

No.

What should a reviewer concentrate their feedback on?

  • Rolling scale estimators: population variance in KL-N/KL-DE1, MAD in KL-DE2/msMAPE
  • Jackknife leave-one-out logic in _evaluate_by_index for NMSE, IQR, KL-N, and TheilU2
  • get_test_params coverage for each metric class

Did you add any tests for the change?

Yes:

  • Individual class tests for all 7 metrics (test_msmape_class, test_nmse_class, test_iqre_class, test_kln_class, test_klde1_class, test_klde2_class, test_theilu2_class)
  • test_evaluate_by_index_returns_correct_index covering all 4 jackknife metrics
  • All 112 framework tests in test_all_metrics_forecasting.py pass

Any other comments?

No.

PR checklist

For all contributions
  • [ X ] I've added myself to the list of contributors with any new badges I've earned :-) How to: add yourself to the [all-contributors fil (https://github.com/sktime/sktime/blob/main/.all-contributorsrc) in the sktime root directory (not the CONTRIBUTORS.md). Common badges: code - fixing a bug, or adding code logic. doc - writing or improving documentation or docstrings. bug - reporting or diagnosing a bug (get this plus code if you also fixed the bug in the PR).maintenance - CI, test framework, release. See here for full badge reference
  • [ X ] Optionally, for added estimators: I've added myself and possibly to the maintainers tag - do this if you want to become the owner or maintainer of an estimator you added. See here for further details on the [algorithm maintainer role (https://www.sktime.net/en/latest/get_involved/governance.html#algorithm-maintainers).
  • [ X ] The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.
For new estimators
  • [ X ] I've added the estimator to the API reference - in docs/source/api_reference/taskname.rst, follow the pattern.
  • [ X ] I've added one or more illustrative usage examples to the docstring, in a pydocstyle compliant Examples section.
  • [ X ] If the estimator relies on a soft dependency, I've set the python_dependencies tag and ensured dependency isolation, see the [estimator dependencies guide (https://www.sktime.net/en/latest/developer_guide/dependencies.html#adding-a-soft-dependency).

@fkiraly fkiraly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

There are a few code formatting issues, could you kindly run pre-commit on your new files?

@fkiraly fkiraly added enhancement Adding new functionality module:metrics&benchmarking metrics and benchmarking modules labels Feb 13, 2026
Add 7 metrics from Chen & Yang (2004): msMAPE, NMSE, IQR, KL-N,
KL-DE1, KL-DE2, and Theil U2. Includes base class for KL-DE variants,
unit tests, API reference docs, and doctest exclusions.
Add __author__ and _tags author attribution to all 7 Chen & Yang
metrics. Improve comments explaining jackknife LOO logic and population
variance computation. Add LaTeX docstring to KL-DE base class. Add
sp > n validation in TheilU2. Add test for evaluate_by_index index
correctness on jackknife metrics.
@michaelellis003

Copy link
Copy Markdown
Contributor Author

Very nice!

There are a few code formatting issues, could you kindly run pre-commit on your new files?

Ran pre-commit. Should be good now. Thanks!

fkiraly
fkiraly previously approved these changes Feb 24, 2026

@fkiraly fkiraly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, thanks!

@fkiraly
fkiraly merged commit 4ab6dec into sktime:main Jul 26, 2026
164 of 187 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Adding new functionality module:metrics&benchmarking metrics and benchmarking modules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ENH] implementing forecasting accuracy metrics from Chen & Yang 2004 "Assessing Forecast Accuracy Measures"

2 participants