Skip to content

ENH: Add VaR backtests (Kupiec, Christoffersen, Basel traffic light) - #965

Merged
bashtage merged 6 commits into
mainfrom
a91-f904-var-backtests
Oct 29, 2026
Merged

bashtage merged 6 commits into
mainfrom
a91-f904-var-backtests

Conversation

@bashtage

@bashtage bashtage commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Target 9.1: merge after v9.0.0 is tagged. xref #904. This implements the five standard
backtests from the proposal. max_undetectable_understatement is left out, pending a
decision on the issue.

Adds the new module arch/univariate/backtest.py, exported from arch.univariate:

  • var_exceptions(returns, var): the exceptions, -returns > var. VaR is a positive
    loss, and a loss equal to the VaR is not an exception. It is a bool array, or a Series
    named "exception" when an input is a pandas object.
  • kupiec_pof(returns, var, p=0.05): the Kupiec (1995) proportion-of-failures
    likelihood-ratio test, chi2(1).
  • christoffersen_independence(returns, var): Christoffersen's (1998) Markov independence
    test, chi2(1).
  • christoffersen_conditional_coverage(returns, var, p=0.05): LR_uc + LR_ind, chi2(2).
  • traffic_light(returns, var, p=0.01, window=250): the Basel Committee (1996) zone,
    returned as a frozen TrafficLightResult with the zone, the count, the cumulative
    probability P(X <= x), the type 1 error P(X >= x) (Table 1 of the framework), nobs
    and p.

The three tests return WaldTestStatistic.

One VaR convention throughout arch: VaR is named by its confidence level, and the
argument is always p, the probability of an exception (the size of the tail): p=0.05
for a 95% VaR and p=0.01 for a 99% VaR. This is rugarch's alpha; MATLAB's VaRLevel
is 1 - p.

  • The tests default to p=0.05, a 95% VaR, as rugarch and MATLAB do. traffic_light
    defaults to the 99% VaR of the Basel framework.
  • var_ratio and ARCHModelFixedResult.var_ratio now take p (default 0.001, a 99.9%
    VaR). Their level, the confidence level, is deprecated with a FutureWarning, and
    p and level cannot both be given.
  • The forecasting notebook labels its VaR columns "99%" and "95%", not "1%" and "5%".

Other conventions:

  • The zones follow the wording of the Basel framework: the yellow zone begins where the
    probability of that number of exceptions or fewer "equals or exceeds 95%", and the red
    zone where it equals or exceeds 99.99%. For other sample sizes the boundaries are
    deduced from the binomial distribution, as the framework says. traffic_light counts
    the last window observations (the Basel year) and raises for a shorter series. Zero
    exceptions is always green.
  • LR_cc is LR_uc on all T observations plus LR_ind on the T-1 transitions, as in rugarch
    and MATLAB.

Inputs:

  • The data must be finite and 1-d.
  • Two Series are aligned on the index. Otherwise the lengths must match, and the inputs
    are matched by position. The docs note that forecasts are indexed by their origin by
    default, so forecast(..., align="target") (or a shift) is needed before a backtest.
  • 0·log 0 = 0 through xlogy.

Validation:

  • R 4.6.1, rugarch 1.5.6 VaRTest, on 11 cases: largest difference in the statistics
    1.2e-14.
  • Degenerate series (zero or all exceptions, first-only or last-only, 2 observations)
    cannot go through rugarch, which indexes a 2x2 table. They are checked against an
    independent R likelihood implementation, cross-checked with rugarch.
  • The traffic-light probabilities and type 1 errors match R pbinom to 1e-12, and the
    published Tables 1 and 2 of the BIS 1996 backtesting framework, including the zone
    boundaries at 5 and 10.
  • The Kupiec 99% non-rejection regions (255/510/1000 observations) match the usually quoted
    values.
  • The references are stored as constants in the tests. The R script that produces them is
    arch/tests/univariate/results/var_backtest_reference.R; it prints the constants, and
    rerunning it reproduces them exactly.

Tests:

  • test_backtest.py and test_validation_backtest.py, plus the var_ratio tests for
    p and the deprecated level.
  • Each of 8 deliberately introduced bugs is caught.
  • The univariate suite passes (14,325 passed), the new tests pass on the minimum versions
    (Python 3.10, NumPy 1.23, SciPy 1.8, pandas 1.5) and with -W error, and the forecasting
    notebook runs. numpydoc lint 0.

For review:

  1. var_ratio(level=...) ships in 9.0, so it is deprecated here rather than removed. If
    this lands before 9.0.0 is tagged, level can be dropped instead.
  2. traffic_light keeps p=0.01 as its default (Basel), while the tests default to 0.05.

Thanks to @sarthakguptaquant for the proposal.

🤖 Generated with Claude Code

Add arch.univariate.backtest with standalone Value-at-Risk backtests, exported from
arch.univariate and added to the utility page of the documentation (xref #904).
VaR is a positive loss and an exception is -returns > var.

- var_exceptions returns the bool array of exceptions.
- kupiec_pof is the proportion-of-failures test of Kupiec (1995), chi2(1).
- christoffersen_independence is the Markov independence test of Christoffersen
  (1998), chi2(1), and christoffersen_conditional_coverage is the sum of the two
  statistics, chi2(2). The tests return a WaldTestStatistic.
- traffic_light is the Basel Committee (1996) zone over the last window observations
  (250 by default) with the exception count and the cumulative binomial probability,
  returned as a TrafficLightResult. For 250 observations and p=0.01 the zones are
  those of Table 2: green 0-4, yellow 5-9, red 10 or more.

The inputs must be finite. Series are aligned on the index when both are Series, and
must otherwise have the same length. A series with no exceptions, only exceptions or no
1 -> 1 transition is handled with the limits of the likelihood (0 log 0 = 0). A series
that is shorter than the window raises.

The tests compare the statistics and p-values with rugarch::VaRTest (1.5.6) and with an
independent likelihood implementation in R, the zones with the binomial distribution
and the published Tables 1 and 2 of the Basel framework, and the non-rejection regions
with Kupiec (1995).
@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.97%. Comparing base (d2c90b4) to head (205f3c6).
⚠️ Report is 40 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #965      +/-   ##
==========================================
+ Coverage   98.95%   98.97%   +0.02%     
==========================================
  Files         136      141       +5     
  Lines       30043    30779     +736     
  Branches     2331     2378      +47     
==========================================
+ Hits        29729    30465     +736     
  Misses        256      256              
  Partials       58       58              
Flag Coverage Δ
adder 98.92% <100.00%> (+0.02%) ⬆️
subtractor 98.92% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bashtage and others added 5 commits October 29, 2026 14:36
The constants in test_backtest.py were produced by a script outside the
repository. Add it as arch/tests/univariate/results/var_backtest_reference.R,
without machine-specific library or output paths: it prints the constants to
stdout. Rerun with R 4.6.1 and rugarch 1.5.6, its output is identical to the
constants in the tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
VaR is named by its confidence level everywhere, and p is always the
probability of an exception: p=0.05 for a 95% VaR and p=0.01 for a 99% VaR.
The notes and the docs said "1% VaR" while the parameters said "99% VaR".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- kupiec_pof and christoffersen_conditional_coverage default to p=0.05, a
  95% VaR, as rugarch (alpha=0.05) and MATLAB (VaRLevel=0.95) do.
  traffic_light keeps the 99% VaR of the Basel framework.
- var_exceptions returns a Series named "exception" when an input is a
  pandas object, with the index of the observations used.
- TrafficLightResult has type1_error, P(X >= x), the type 1 error of Table 1
  of the Basel framework, checked against R pbinom.
- The notes cite the Basel wording for the zone boundaries ("equals or
  exceeds" 95% and 99.99%) and say that only two Series are aligned on the
  index, and that forecasts are indexed by their origin by default.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
VaR is named by its confidence level, and the argument is always p, the
probability of an exception: p=0.05 for a 95% VaR.

- var_ratio and ARCHModelFixedResult.var_ratio take p (default 0.001, a
  99.9% VaR). level, the confidence level, is deprecated with a
  FutureWarning, and p and level cannot both be given.
- The forecasting notebook labels its VaR columns "99%" and "95%", not
  "1%" and "5%".
- The docs describe the convention, the defaults and the alignment of
  forecasts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Codacy (pylint C0123) flags type(x) is int. Check for a plain Python int
and float with isinstance, and exclude bool and NumPy scalars explicitly,
since np.float64 is a subclass of float. Also check type1_error.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@bashtage
bashtage merged commit a448147 into main Oct 29, 2026
26 checks passed
@bashtage
bashtage deleted the a91-f904-var-backtests branch October 29, 2026 16:33
jngod2011 pushed a commit to jngod2011/arch that referenced this pull request Oct 31, 2026
- Highlights and new features: NGARCH (bashtage#966), the VaR backtests (bashtage#965), the
  local block bootstrap (bashtage#967) and bandwidth_method (bashtage#964, and
  phillips_ouliaris here).
- Changes that alter results: the automatic bandwidth entry of bashtage#983 now
  also describes the bandwidth of the kernels with bounded support, the
  fractional bandwidths and the Tukey-Parzen constant of bashtage#964, its S&P 500
  bandwidths are those of the bandwidth attribute (30.96 and 34.50, where it
  said the 31.96 and 35.50 of opt_bandwidth), and it links phillips_ouliaris
  in arch.unitroot.cointegration. The ADF-Z tables of bashtage#972, bashtage#973 and bashtage#980.
- Backward-incompatible changes: strings in object data.
- Bug fixes: bashtage#954, bashtage#970, bashtage#974, bashtage#975, bashtage#977, bashtage#978, bashtage#982, and the constant
  data of the Newey-West bandwidth and mackinnonp at -inf and inf.
- Maintenance: the CI changes and the version file.
- Contributors: galafis, jackiectl2, mkzung and wavebyrd.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants