Repository navigation
ENH: Add VaR backtests (Kupiec, Christoffersen, Basel traffic light) - #965
Merged
Merged
Conversation
Add arch.univariate.backtest with standalone Value-at-Risk backtests, exported from arch.univariate and added to the utility page of the documentation (xref #904). VaR is a positive loss and an exception is -returns > var. - var_exceptions returns the bool array of exceptions. - kupiec_pof is the proportion-of-failures test of Kupiec (1995), chi2(1). - christoffersen_independence is the Markov independence test of Christoffersen (1998), chi2(1), and christoffersen_conditional_coverage is the sum of the two statistics, chi2(2). The tests return a WaldTestStatistic. - traffic_light is the Basel Committee (1996) zone over the last window observations (250 by default) with the exception count and the cumulative binomial probability, returned as a TrafficLightResult. For 250 observations and p=0.01 the zones are those of Table 2: green 0-4, yellow 5-9, red 10 or more. The inputs must be finite. Series are aligned on the index when both are Series, and must otherwise have the same length. A series with no exceptions, only exceptions or no 1 -> 1 transition is handled with the limits of the likelihood (0 log 0 = 0). A series that is shorter than the window raises. The tests compare the statistics and p-values with rugarch::VaRTest (1.5.6) and with an independent likelihood implementation in R, the zones with the binomial distribution and the published Tables 1 and 2 of the Basel framework, and the non-rejection regions with Kupiec (1995).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #965 +/- ##
==========================================
+ Coverage 98.95% 98.97% +0.02%
==========================================
Files 136 141 +5
Lines 30043 30779 +736
Branches 2331 2378 +47
==========================================
+ Hits 29729 30465 +736
Misses 256 256
Partials 58 58
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The constants in test_backtest.py were produced by a script outside the repository. Add it as arch/tests/univariate/results/var_backtest_reference.R, without machine-specific library or output paths: it prints the constants to stdout. Rerun with R 4.6.1 and rugarch 1.5.6, its output is identical to the constants in the tests. Co-Authored-By: Claude Opus 5.5 <[email protected]>
VaR is named by its confidence level everywhere, and p is always the probability of an exception: p=0.05 for a 95% VaR and p=0.01 for a 99% VaR. The notes and the docs said "1% VaR" while the parameters said "99% VaR". Co-Authored-By: Claude Opus 5.5 <[email protected]>
- kupiec_pof and christoffersen_conditional_coverage default to p=0.05, a
95% VaR, as rugarch (alpha=0.05) and MATLAB (VaRLevel=0.95) do.
traffic_light keeps the 99% VaR of the Basel framework.
- var_exceptions returns a Series named "exception" when an input is a
pandas object, with the index of the observations used.
- TrafficLightResult has type1_error, P(X >= x), the type 1 error of Table 1
of the Basel framework, checked against R pbinom.
- The notes cite the Basel wording for the zone boundaries ("equals or
exceeds" 95% and 99.99%) and say that only two Series are aligned on the
index, and that forecasts are indexed by their origin by default.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
VaR is named by its confidence level, and the argument is always p, the probability of an exception: p=0.05 for a 95% VaR. - var_ratio and ARCHModelFixedResult.var_ratio take p (default 0.001, a 99.9% VaR). level, the confidence level, is deprecated with a FutureWarning, and p and level cannot both be given. - The forecasting notebook labels its VaR columns "99%" and "95%", not "1%" and "5%". - The docs describe the convention, the defaults and the alignment of forecasts. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Codacy (pylint C0123) flags type(x) is int. Check for a plain Python int and float with isinstance, and exclude bool and NumPy scalars explicitly, since np.float64 is a subclass of float. Also check type1_error. Co-Authored-By: Claude Opus 5.5 <[email protected]>
jngod2011
pushed a commit
to jngod2011/arch
that referenced
this pull request
Oct 31, 2026
- Highlights and new features: NGARCH (bashtage#966), the VaR backtests (bashtage#965), the local block bootstrap (bashtage#967) and bandwidth_method (bashtage#964, and phillips_ouliaris here). - Changes that alter results: the automatic bandwidth entry of bashtage#983 now also describes the bandwidth of the kernels with bounded support, the fractional bandwidths and the Tukey-Parzen constant of bashtage#964, its S&P 500 bandwidths are those of the bandwidth attribute (30.96 and 34.50, where it said the 31.96 and 35.50 of opt_bandwidth), and it links phillips_ouliaris in arch.unitroot.cointegration. The ADF-Z tables of bashtage#972, bashtage#973 and bashtage#980. - Backward-incompatible changes: strings in object data. - Bug fixes: bashtage#954, bashtage#970, bashtage#974, bashtage#975, bashtage#977, bashtage#978, bashtage#982, and the constant data of the Newey-West bandwidth and mackinnonp at -inf and inf. - Maintenance: the CI changes and the version file. - Contributors: galafis, jackiectl2, mkzung and wavebyrd.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Target 9.1: merge after v9.0.0 is tagged. xref #904. This implements the five standard
backtests from the proposal.
max_undetectable_understatementis left out, pending adecision on the issue.
Adds the new module
arch/univariate/backtest.py, exported fromarch.univariate:var_exceptions(returns, var): the exceptions,-returns > var. VaR is a positiveloss, and a loss equal to the VaR is not an exception. It is a bool array, or a Series
named
"exception"when an input is a pandas object.kupiec_pof(returns, var, p=0.05): the Kupiec (1995) proportion-of-failureslikelihood-ratio test, chi2(1).
christoffersen_independence(returns, var): Christoffersen's (1998) Markov independencetest, chi2(1).
christoffersen_conditional_coverage(returns, var, p=0.05): LR_uc + LR_ind, chi2(2).traffic_light(returns, var, p=0.01, window=250): the Basel Committee (1996) zone,returned as a frozen
TrafficLightResultwith the zone, the count, the cumulativeprobability P(X <= x), the type 1 error P(X >= x) (Table 1 of the framework),
nobsand
p.The three tests return
WaldTestStatistic.One VaR convention throughout arch: VaR is named by its confidence level, and the
argument is always
p, the probability of an exception (the size of the tail):p=0.05for a 95% VaR and
p=0.01for a 99% VaR. This is rugarch'salpha; MATLAB'sVaRLevelis
1 - p.p=0.05, a 95% VaR, as rugarch and MATLAB do.traffic_lightdefaults to the 99% VaR of the Basel framework.
var_ratioandARCHModelFixedResult.var_rationow takep(default 0.001, a 99.9%VaR). Their
level, the confidence level, is deprecated with aFutureWarning, andpandlevelcannot both be given.Other conventions:
probability of that number of exceptions or fewer "equals or exceeds 95%", and the red
zone where it equals or exceeds 99.99%. For other sample sizes the boundaries are
deduced from the binomial distribution, as the framework says.
traffic_lightcountsthe last
windowobservations (the Basel year) and raises for a shorter series. Zeroexceptions is always green.
and MATLAB.
Inputs:
are matched by position. The docs note that forecasts are indexed by their origin by
default, so
forecast(..., align="target")(or a shift) is needed before a backtest.0·log 0 = 0throughxlogy.Validation:
VaRTest, on 11 cases: largest difference in the statistics1.2e-14.
cannot go through rugarch, which indexes a 2x2 table. They are checked against an
independent R likelihood implementation, cross-checked with rugarch.
pbinomto 1e-12, and thepublished Tables 1 and 2 of the BIS 1996 backtesting framework, including the zone
boundaries at 5 and 10.
values.
arch/tests/univariate/results/var_backtest_reference.R; it prints the constants, andrerunning it reproduces them exactly.
Tests:
test_backtest.pyandtest_validation_backtest.py, plus thevar_ratiotests forpand the deprecatedlevel.(Python 3.10, NumPy 1.23, SciPy 1.8, pandas 1.5) and with
-W error, and the forecastingnotebook runs. numpydoc lint 0.
For review:
var_ratio(level=...)ships in 9.0, so it is deprecated here rather than removed. Ifthis lands before 9.0.0 is tagged,
levelcan be dropped instead.traffic_lightkeepsp=0.01as its default (Basel), while the tests default to 0.05.Thanks to @sarthakguptaquant for the proposal.
🤖 Generated with Claude Code