Repository navigation
Studentize loss differentials in SPA and StepM when studentize=True - #871
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #871 +/- ##
=======================================
Coverage 99.54% 99.54%
=======================================
Files 78 78
Lines 15847 15874 +27
Branches 1296 1297 +1
=======================================
+ Hits 15775 15802 +27
Misses 38 38
Partials 34 34
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Owner
|
Thanks. |
This was referenced Oct 7, 2026
bashtage
added a commit
to shaneraphel/arch
that referenced
this pull request
Oct 7, 2026
full-sample standard error. The observed statistic then contains the sampling error of its estimated standard error, but the bootstrap distribution does not, so SPA and StepM over-reject. In a Monte Carlo (AR(1) loss differentials with phi=0.5 and a common factor, a 100-fold range of model volatilities, 1000 trials, 399 bootstrap replications, nominal 5%), the consistent p-value rejected a true null in 14.6-18.5% of trials at T=250, k=20 and 19.3-25.0% at T=100, k=10. With this change it rejects in 5.6-6.1% and 5.4-6.8% of trials. Not studentizing, the behavior before bashtage#871, gave 6.6-7.3% and 10.5-12.0%. The ranges cover the stationary, circular and moving block bootstraps with a block size of 10. Each replication now computes the standard error of every column from the bootstrap sample, and divides the re-centered mean by it, as in the studentized block bootstrap of Gotze and Kunsch (1996). A nested bootstrap for every replication is far too slow, so the standard error uses a kernel long-run variance estimator with the kernel implied by the bootstrap: * Stationary bootstrap: the weights (1-i/T)(1-p)^i + (i/T)(1-p)^(T-i) of Politis and Romano (1994), which give the exact bootstrap variance of the mean. * Circular block bootstrap: the Bartlett kernel on the circular autocovariances, adjusted for the final truncated block, which gives the exact bootstrap variance of the mean. * Moving block bootstrap: the Bartlett kernel, which equals the bootstrap variance of the mean up to edge effects. A column that is constant in a bootstrap sample has a zero standard error. This happens when a model differs from the benchmark in very few periods, for example with 0/1 losses. Dividing by it gave statistics of order 1e13 in about a third of the replications for a model that differs from the benchmark in a single period, which made the critical values huge and removed the power to detect any other model. In those replications the full-sample standard error is used for that column. The estimator is computed for all columns with one FFT, so it costs O(T log T) per column and replication. The full-sample variance used for the observed statistic and for the consistent recentering threshold uses the same kernel. Previously it always used the stationary bootstrap kernel, so the circular and moving block bootstraps give different variances, and so different p-values, than before. nested=True still only changes the full-sample variance; the standard errors inside the bootstrap always use the kernel. Tests compare the kernels with the exact bootstrap variance of the mean (the Markov chain of the stationary bootstrap, full enumeration of the circular block bootstrap, the exact moving block bootstrap variance and arch.covariance.kernel.Bartlett), the FFT estimator with a direct sum of autocovariances, and the simulated statistics with a replication-by- replication recomputation from the same bootstrap draws.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SPA(..., studentize=True)is the default, and Hansen's SPA test uses the studentized statistic, but the flag currently only changes the label in_info.compute()compares the rawloss_diff.mean()with raw simulated means, andbetter_models()compares raw means with critical values, sostudentize=Trueandstudentize=Falsegive identical results.This matters when models differ in how noisy their losses are. With a benchmark and two models, one slightly better with tiny noise and one better on average with huge noise (500 obs, seed 0,
block_size=10,reps=500):The fix divides both the sample mean loss differentials and the simulated values by their standard errors,
sqrt(loss_diff_var / t), whenstudentize=True.studentize=Falseis unchanged. The standard error is floored at machine epsilon to avoid dividing by zero.Since
studentize=Trueis the default, p-values and critical values from default settings will change. I have noted in thecritical_valuesdocstring that with studentization they apply to the studentized means.Tests:
test_studentizationinarch/tests/bootstrap/test_multiple_comparison.pychecks the p-values against a hand-studentized version of the unstudentized run, and that the precise model is selected. It fails on main and passes with the fix.test_pvalues_and_critvalsbuilt its reference from the raw means, so its reference now divides by the standard error.arch/tests/bootstrap/test_multiple_comparison.pypasses (28 passed), andblack,isortandruffare clean.