Skip to content

ENH: VaR backtests (Kupiec, Christoffersen, Basel traffic light) and a "maximum undetectable understatement" diagnostic #904

Open

Description

@sarthakguptaquant

arch has tools for fitting models that produce VaR forecasts and for forecasting them, but nothing that backtests a VaR series against realized returns. I searched the current main branch for "backtest", "kupiec" and "christoffersen" and found no matches.

I would like to add a small set of standalone functions in arch.univariate, in the same shape as the residual diagnostics added in #850:

from arch.univariate import (
    var_exceptions,                       # bool array: -returns > var
    kupiec_pof,                           # LR test, WaldTestStatistic (df=1)
    christoffersen_independence,          # LR test, WaldTestStatistic (df=1)
    christoffersen_conditional_coverage,  # LR test, WaldTestStatistic (df=2)
    traffic_light,                        # Basel zone for the last 250 days
    max_undetectable_understatement,
)

The first five are textbook. The three likelihood-ratio tests return the existing WaldTestStatistic, so .stat, .pval and .critical_values work as they do elsewhere in the package; var_exceptions returns a boolean array and traffic_light returns a small result object with the zone, the count and the cumulative probability. The Kupiec non-rejection regions match the ones usually quoted from Kupiec (1995) for 99% VaR, and the traffic light cumulative probabilities reproduce Table 2 of the Basel Committee's 1996 backtesting framework.

The sixth function is the part I think is new. A passing backtest says the reported VaR was not rejected. It does not say how close it was. max_undetectable_understatement(returns, var) answers a concrete question: by what fraction could the whole VaR series have been scaled down, with the same returns, and still pass every selected test? The result also reports the extra exceptions that scaling would produce and which test binds first. A pass with a limit of 2% reads very differently from a pass with a limit of 25%.

Because the exception set only changes at the points 1 - loss/var, the search over proportional scalings is exact and needs one pass over the breakpoints, no optimizer. It is a lower bound on what a non-proportional series could hide, and the docstring says so. The tests include a brute-force grid check of the limit, and it also agrees with a bisection solver I wrote earlier on 34 simulated series to about 1e-16.

Three design points I would like your view on:

  1. Placement. I put it in arch/univariate/backtest.py next to diagnostics.py, with docs under the utility page. If you would rather have it elsewhere, moving it is trivial.

  2. Kupiec is two-sided, so a bank that reports zero exceptions fails it for being too conservative. For the understatement question that is not informative, so max_undetectable_understatement ignores rejections caused by too few exceptions by default (ignore_too_few=True).

  3. Wrapper. On ENH: Add tail risk diagnostics to ARCHModelResult (Hill estimator, excess kurtosis) #845 you said you would rather have standalone functions with a lightweight method wrapper. These are standalone, and the first five would wrap onto ARCHModelResult cleanly once a VaR series is supplied. I left the wrapper out of the sketch so the functions can be reviewed on their own, and will add it if you want it.

I have a working branch with tests (hand-computed Christoffersen values, the published Kupiec and Basel tables, property tests and a brute-force grid check of the understatement limit) and docs. I am happy to open a PR with the whole thing, or with only the five standard tests first if you prefer to review the new diagnostic separately. If this is out of scope for arch, I will publish it as a small standalone package instead.

Activity

  1. added a commit that references this issue on Oct 31, 2026
  2. sarthakguptaquant commented on Nov 8, 2026

    @sarthakguptaquant
    Author

    Thank you for implementing the five tests in #965, and for the credit.

    On max_undetectable_understatement, which was left out pending a decision: the branch with the function, docs and the brute-force grid check of the limit is ready, and I would adapt it to the conventions you used in backtest.py. If you would like it in arch, I will open it as a small follow-up PR against main. If you would rather keep the module to the standard tests, that is completely fine, and I will release it as a standalone package that works with arch VaR output. Happy to go either way.

  3. bashtage commented on Nov 8, 2026

    @bashtage
    Owner

    Happy to have a follow up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions