Repository navigation
SPA with studentize=True (default) is oversized after #871 at the default block length #879
Description
Thanks for merging #871. Before it is released, here is a measurement of the merged code that may help.
What I ran: arch at merge commit 818d64a (8.0.1.dev83). SPA(np.zeros(504), -x, bootstrap="stationary", reps=1000) with default settings (studentize=True, block_size=int(sqrt(504))=22). I ran it on 2,000 simulated searches per family, each with 20 strategies, 504 observations and every true mean zero. I counted how often the consistent p-value was at most 0.05.
| Returns | arch 8.0.0 | arch at 818d64a | arch at 818d64a, block_size=8 |
|---|---|---|---|
| i.i.d. normal | 6.5% | 13.5% | 8.3% |
| negatively skewed | 5.5% | 19.1% | 13.4% |
| AR(1), phi = 0.2 | 7.4% | 13.3% | 9.3% |
The cause: the studentization divides both the observed and the bootstrapped means by the same sample standard error, sqrt(loss_diff_var / t). The bootstrap therefore ignores that estimate's sampling noise, and the estimate (from demeaned data) is biased downward. Romano and Wolf (2005, Econometrica) note that holding it fixed is only first-order accurate and recommend re-estimating it in each resample. In our simulations, re-studentizing every resample by its own standard deviation (a bootstrap-t) kept the size at 5.0–6.2% across eight return shapes at block length 8.
One possible fix: compute each bootstrap replication's standard error from that resample and divide its centred mean by it.
Code, results and pre-registration: https://github.com/arhancanli/null-zoo (branch v2). The check script is analysis/v2/arch_merged_check.py and its output is analysis/v2/arch-merged-check.json.
Activity
bashtage commentedon Oct 7, 2026
Can't do a nested bootstrap as it is too expensive. Could use a closed-form estimator à la Hansen's paper within each bootstrap.
ENH: Studentize each bootstrap resample in SPA and StepM (bootstrap-t)
arhancanli commentedon Oct 7, 2026
Thanks. What we ran didn't need a nested bootstrap: each replication's centred mean is divided by that resample's own standard deviation (over sqrt(T)), computed from the resampled series itself. That is one extra pass over the resample per replication, the same order of cost as taking the means.
A Hansen-style closed-form estimator of the variance inside each replication would be the more faithful version for dependent data, since a plain sd ignores the block dependence. We only tested the plain sd, so I can't say yet how the two compare under the AR(1) case. If that's the direction you'd prefer, I can rerun the same return shapes with a per-resample estimator of that kind and post the size table here.
bashtage commentedon Oct 9, 2026
This appears to have been fixed in #882