Skip to content

Studentize loss differentials in SPA and StepM when studentize=True - #871

Merged
bashtage merged 1 commit into
bashtage:mainfrom
WatchTree-19:fix/spa-studentize
Oct 6, 2026
Merged

bashtage merged 1 commit into
bashtage:mainfrom
WatchTree-19:fix/spa-studentize

Conversation

@WatchTree-19

Copy link
Copy Markdown
Contributor

SPA(..., studentize=True) is the default, and Hansen's SPA test uses the studentized statistic, but the flag currently only changes the label in _info. compute() compares the raw loss_diff.mean() with raw simulated means, and better_models() compares raw means with critical values, so studentize=True and studentize=False give identical results.

This matters when models differ in how noisy their losses are. With a benchmark and two models, one slightly better with tiny noise and one better on average with huge noise (500 obs, seed 0, block_size=10, reps=500):

                    lower   consistent  upper   better_models
main, True          0.336   0.426       0.426   []
main, False         0.336   0.426       0.426
fixed, True         0.000   0.000       0.000   [0]
StepM superior:     main []   fixed [0]

The fix divides both the sample mean loss differentials and the simulated values by their standard errors, sqrt(loss_diff_var / t), when studentize=True. studentize=False is unchanged. The standard error is floored at machine epsilon to avoid dividing by zero.

Since studentize=True is the default, p-values and critical values from default settings will change. I have noted in the critical_values docstring that with studentization they apply to the studentized means.

Tests:

  • New test_studentization in arch/tests/bootstrap/test_multiple_comparison.py checks the p-values against a hand-studentized version of the unstudentized run, and that the precise model is selected. It fails on main and passes with the fix.
  • test_pvalues_and_critvals built its reference from the raw means, so its reference now divides by the standard error.
  • arch/tests/bootstrap/test_multiple_comparison.py passes (28 passed), and black, isort and ruff are clean.

@codecov

codecov Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.54%. Comparing base (88ffcf8) to head (6f1ca7f).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #871   +/-   ##
=======================================
  Coverage   99.54%   99.54%           
=======================================
  Files          78       78           
  Lines       15847    15874   +27     
  Branches     1296     1297    +1     
=======================================
+ Hits        15775    15802   +27     
  Misses         38       38           
  Partials       34       34           
Flag Coverage Δ
adder 99.50% <100.00%> (+<0.01%) ⬆️
subtractor 99.50% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bashtage
bashtage merged commit 818d64a into bashtage:main Oct 6, 2026
27 checks passed
@bashtage

bashtage commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Thanks.

bashtage added a commit to shaneraphel/arch that referenced this pull request Oct 7, 2026
full-sample standard error. The observed statistic then contains the
sampling error of its estimated standard error, but the bootstrap
distribution does not, so SPA and StepM over-reject. In a Monte Carlo
(AR(1) loss differentials with phi=0.5 and a common factor, a 100-fold
range of model volatilities, 1000 trials, 399 bootstrap replications,
nominal 5%), the consistent p-value rejected a true null in 14.6-18.5% of
trials at T=250, k=20 and 19.3-25.0% at T=100, k=10. With this change it
rejects in 5.6-6.1% and 5.4-6.8% of trials. Not studentizing, the
behavior before bashtage#871, gave 6.6-7.3% and 10.5-12.0%. The ranges cover the
stationary, circular and moving block bootstraps with a block size of 10.

Each replication now computes the standard error of every column from the
bootstrap sample, and divides the re-centered mean by it, as in the
studentized block bootstrap of Gotze and Kunsch (1996). A nested bootstrap
for every replication is far too slow, so the standard error uses a kernel
long-run variance estimator with the kernel implied by the bootstrap:

 * Stationary bootstrap: the weights (1-i/T)(1-p)^i + (i/T)(1-p)^(T-i) of
   Politis and Romano (1994), which give the exact bootstrap variance of
   the mean.
 * Circular block bootstrap: the Bartlett kernel on the circular
   autocovariances, adjusted for the final truncated block, which gives
   the exact bootstrap variance of the mean.
 * Moving block bootstrap: the Bartlett kernel, which equals the
   bootstrap variance of the mean up to edge effects.

A column that is constant in a bootstrap sample has a zero standard
error. This happens when a model differs from the benchmark in very few
periods, for example with 0/1 losses. Dividing by it gave statistics of
order 1e13 in about a third of the replications for a model that differs
from the benchmark in a single period, which made the critical values
huge and removed the power to detect any other model. In those
replications the full-sample standard error is used for that column.

The estimator is computed for all columns with one FFT, so it costs
O(T log T) per column and replication. The full-sample variance used for
the observed statistic and for the consistent recentering threshold uses
the same kernel. Previously it always used the stationary bootstrap
kernel, so the circular and moving block bootstraps give different
variances, and so different p-values, than before. nested=True still
only changes the full-sample variance; the standard errors inside the
bootstrap always use the kernel.

Tests compare the kernels with the exact bootstrap variance of the mean
(the Markov chain of the stationary bootstrap, full enumeration of the
circular block bootstrap, the exact moving block bootstrap variance and
arch.covariance.kernel.Bartlett), the FFT estimator with a direct sum of
autocovariances, and the simulated statistics with a replication-by-
replication recomputation from the same bootstrap draws.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants