Skip to content

About

Market regimes from correlation structure: random-matrix spectra, correlation networks, out-of-sample HMM, and QAOA community detection

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

market-states-mqfs

Market states from correlation structure: detection without timing.

A Mediterranean Quantitative Finance Society (MQFS) working paper.

Author: Nicolò Angileri (ORCID 0009-0004-3971-1114) · MIT licence

A correlation state detects crises reliably and cannot time them. This repository establishes both halves of that sentence on one empirical object, in four steps that share code, windows and seeds:

  1. What is really there — rolling 1-year correlation matrices, Marchenko–Pastur eigenvalue clipping (Laloux et al. 1999), Mantegna minimum spanning trees (Mantegna 1999), with dependence-aware moving-block bootstrap inference and seeded 3-D Monte Carlo checks.
  2. Does it move in states — a hand-written, brute-force-tested Gaussian HMM on seven per-window spectral/network features, a pre-registered BIC-then-silhouette state-count rule, a strictly frozen out-of-sample protocol, and a returns-only Markov-switching cross-check.
  3. Is it worth anything — the frozen decoder drives exposure against the control a practitioner already has (a flat book de-levered to the same volatility), under four designs, then replicated across 5 markets whose returns correlate as low as 0.072 and 16 disjoint (market, era) cells reaching back to 1940. It loses essentially everywhere, and the exceptions are named.
  4. The combinatorics inside it (a chapter, not a co-equal lens — about a quarter of the paper) — two-community modularity maximisation on thresholded sub-networks mapped exactly to Ising (the linear field vanishes identically — proved and unit-tested), the MacMahon–Garlaschelli correlation null, and a K=3 Potts encoding on 18 exactly-enumerated qubits, solved with QAOA and benchmarked against classical baselines.

No quantum advantage is claimed. The instances are classically easy by construction; the contribution is the exact, tested mapping and the measured behaviour, wall-clock included. The financial argument runs uninterrupted from correlations to the economic verdict; the combinatorial chapter comes after it, so a reader who came for the market states can stop before it without missing an argument.

📄 The working paper (48 pages): paper/paper.pdf — every number in it is interpolated at build time from results/*.json.


Data

Dataset: Kenneth French, 49 Industry Portfolios (daily, value-weighted) — fetched once and cached by market_states.french (defensive parser: text preamble, multiple blocks per file, -99.99/-999 missing markers, ragged-row detection; 16 unit tests on synthetic fixtures). The archived run below is the official 49-industry file (202606 CRSP vintage).

.venv/bin/python experiments/run_all.py

regenerates everything — JSON results, all figures, the PDF — in a few minutes plus the LaTeX build.

Earlier version. Releases up to v2 were archived on the 10-industry file, because the sandbox that produced them could not reach the Dartmouth server. That run is tagged v2 and its paper is kept at paper/archive/paper_v2_10industry.pdf. Moving to the full panel changes numbers and one conclusion — the pre-registered rule now selects K = 2 rather than K = 3; see CHANGELOG.md.

Headline numbers (archived run, 1994-01-03 → 2026-06-30, 8,177 days, 378 windows, N = 49)

lens result
RMT market-mode share 0.547 (calm) → 0.682 (crisis); 2 eigenvalues clear λ₊ in both snapshot windows; bulk variance share 40.4% → 18.4%; λ₊ = 2.076 at q = 0.1944
networks mean correlation 0.508 → 0.751; MST length 35.55 → 22.15 (crisis tree contracts by 38%)
regimes K = 2 (silhouette decides — BIC has no interior minimum under overlapping windows); stress state: 36% occupancy, crisis recall 92.9%, precision 19.1%, lift 2.6×; HMM↔GMM agreement ARI 0.64
out-of-sample trained through 2012-12-31 (K, standardisation, all parameters frozen): 80% of COVID windows flagged as stress, 2020 never seen
inference moving-block bootstrap (block 12, 2,000 reps): 5 of 7 calm/crisis contrasts exclude zero — every one the thesis rests on (mean correlation, market-mode share, noise fraction, MST length, dispersion); the two that do not, n_signal and lambda2_share, were artefacts of the 10-asset panel, see CHANGELOG.md — returns-only Markov-switching cross-check ARI 0.32
economics frozen decoder driving exposure: Sharpe 0.57 vs 0.80 for the panel, and vs 0.80 for a flat book at the same volatility — the timing decision gives up 3.1%/yr and buys nothing; it does cut max drawdown 38.4% → 30.7%
panel width same pipeline at N = 49 / 95 / 280 / 423: the calm-vs-crisis correlation signal holds everywhere, but the selected K rises from 2 to 6 with N, and the overlay's loss grows and becomes significant (bootstrap CI excludes zero at N ≥ 95)
QAOA (p=3) merit 0.51 and P(optimum) 0.070 on the largest instance (n=12); brute force 0.24 ms; XY-mixer in-constraint probability 1.000 vs 0.719 for the quadratic penalty

Does the regime label pay for itself?

No — and the answer is the point.

The decoder frozen at 2012-12-31 drives exposure on the equal-weight panel, 5 bp a switch, cash earning nothing, scored only on 2013-01-08 → 2026-06-30. Four designs, so the verdict is about the label and not about one crude way of using it:

design Sharpe return vol max DD vs flat book at same vol
binary on/off 0.571 7.9% 13.8% −30.7% −0.225
volatility target, no regime 0.689 12.2% 17.7% −30.8% −0.090
state-conditional exposure 0.761 12.5% 16.4% −34.3% −0.035
volatility target + regime 0.662 11.6% 17.5% −29.2% −0.117

Three readings. Nothing beats the flat book — the gentler the use of the label, the smaller the loss, but it never changes sign. The label subtracts from what volatility already says: vol targeting alone scores 0.689, and multiplying it by the regime exposure takes it down to 0.662. But vol targeting loses too, so the claim is "the regime label is the worst of the risk-scaling rules and degrades the standard one", not "it is uniquely useless".

The mechanism is legible in the spells: the COVID spell opens 2020-03-12 and closes 2021-04-12. The rule de-risks days after the bottom and stays out through the recovery. Correlation tightens as a crisis breaks and stays tight long after prices recover — a good crisis detector, a bad exit-and-re-entry rule.

Does it survive a wider cross-section? (N = 49 → 423)

The description does, the state count does not, and the overlay's failure is not an artefact of N: the Sharpe gap's bootstrap interval contains zero at N = 49 but excludes it at N = 95 and N = 280.

Two limits we impose on ourselves rather than leave to the reader:

  • The panels are not four independent tests. Same market, same years, different cross-sections: their equal-weight return series correlate 0.947–0.997. They show the result is not a peculiarity of 49 industry portfolios; they do not multiply the evidence.
  • The survivorship bias is measured, not just confessed. Over 2003-05-02 → 2026-08-19 the panel of today's S&P 500 members returns 16.6%/yr against 12.8% for the traded equal-weight index: 3.9 points a year and 0.220 of Sharpe. It is almost entirely in the level — the two daily series correlate 0.9891 — so the co-movement structure studied here is barely touched, and the overlay and its control share the same panel, so it cancels there too.

What survives across all four panels is not "the market has K states" but a monotone stress ordering along mean correlation whose top end coincides with crises. K itself moves from 2 to 6 with N.

Does it replicate outside the US?

Yes — and this is the only part of the paper that buys real independence.

The scale panels above are the same US market four ways (returns correlating 0.947–0.997). So the pipeline runs unchanged on Kenneth French's regional daily portfolios — Europe, Japan, Asia Pacific ex Japan, North America as a control — plus current-constituent panels where French has no coverage.

market N K OOS recall benchmark overlay flat, same vol given up
Europe 25 4 80% 0.57 0.18 0.57 5.07%
Japan 25 3 100% 0.57 0.42 0.57 1.73%
Asia Pacific ex Japan 25 3 80% 0.33 0.01 0.33 3.98%
North America 25 2 100% 0.66 0.12 0.66 5.51%
Israel (TA-35) † 10 5 100% 1.09 0.60 1.09 11.89%

The binary overlay beats the flat book in 0 of 5, giving up 5.6% a year on average. And these markets' returns correlate only 0.072–0.799 — Japan and North America are close to unrelated — against 0.947–0.997 for the US panels. That is the difference between one result seen repeatedly and several results that agree.

Two markets are reported and excluded. Australia (ASX 200) and the UK (FTSE 100) have frozen decoders that never land on a known crisis: recall 0%, one stress spell outside COVID. A rule driven by a decoder that did not fire is not a test of the rule. Both would have flattered us — their overlays sit level with or above the flat book — and both are excluded for it. The cause is worth keeping: those panels start in 2005, so their training half is dominated by the GFC and COVID does not clear the bar. The frozen protocol is only as good as the variety in its training window — a fragility the US-only version of this work could not have seen.

Africa is missing, and stays missing: no Johannesburg constituent list we could reach parses reliably, and inventing one would be worse than the gap.

Does it replicate in another era?

The markets section replicated across space. This replicates across time — the US industry panel starts in 1926, so the frozen protocol can be walked forward through the century.

Two rules make it honest. The eras are disjoint (each cell stops where the next one's training starts) — nested windows would repeat the very error this paper charges against its own scale annex, and nesting produced a suspiciously clean sweep that disjoint windows do not. And crises are identified mechanically (the market fell >20% inside the window), because a calendar of named episodes is useless for 1930 or 1974 and lets the author pick dates after seeing results.

16 disjoint (market, era) cells, earliest evaluated from 1940-01-10. The overlay beats the vol-matched flat book in 3, median give-up 3.4%/yr, p = 0.011 against a coin-flip null.

The three exceptions, named rather than buried: US 1940–54 (+0.03 Sharpe), US 1985–99 (+0.13), Japan 2006–12 (+0.15). All three bootstrap intervals contain zero, and the Japanese cell has a negative benchmark Sharpe — in a market that spent the period falling, holding less is not skill.

Two cells were excluded for degenerate decoders: one that never called stress and one that called it always. Neither varies exposure, so neither measures timing.

Established: the failure holds across markets whose returns barely move together and across eras separated by decades. Not established: anything about correlation states used for something other than sizing beta — cross-sectional allocation, hedging, risk-model conditioning. This paper closes one door firmly and says nothing about the others.

Quickstart

uv venv --python 3.12 .venv
uv pip install -e ".[dev,hardware]"
.venv/bin/python -m pytest -q          # known-answer test suite, no network
.venv/bin/python experiments/run_all.py  # parts 1–3 + the PDF

Single seed (7) drives every stochastic component; re-runs bit-reproduce the JSON payloads up to wall-clock fields. CI runs the suite on CPython 3.10 and 3.12. (The archived run itself executed on CPython 3.10.12 because the sandbox proxy blocks the 3.12 standalone download — noted in the paper.)

Optional: one real-hardware point

The IBM runner is built, mock-tested and deliberately not executed here: run python experiments/hardware.py locally once, where ~/.qiskit holds the IBM account. It rebuilds the optimised p=3 circuit for the stored n=6 instance, transpiles at optimization_level=3 for the least-busy backend, and runs SamplerV2 at 8192 shots with XY4 dynamical decoupling + Pauli twirling; --submit-only / --job-id <id> decouple submission from retrieval. Results land in results/hardware.json + paper/figures/fig_hardware.pdf, and the paper picks the figure up automatically. Credentials are read exclusively via QiskitRuntimeService(); nothing credential-shaped can enter the repo (.gitignore excludes token-like patterns).

Layout

src/market_states/   french.py rmt.py network.py features.py hmm.py
                     regimes.py qubo.py qaoa.py classical.py hardware.py
                     communities.py (MG null + Potts) bootstrap.py montecarlo.py
tests/               116 known-answer tests (parser quirks, MP bounds,
                     QUBO<->Ising / MG / Potts by exhaustive enumeration,
                     HMM vs brute force, QAOA amplitudes vs Qiskit to 1e-10,
                     block bootstrap on planted shifts, MC vs analytic law,
                     mocked sampler)
experiments/         run_econophysics.py run_regimes.py run_montecarlo.py
                     run_quantum.py run_all.py hardware.py
results/             archived *.json + features.csv (every paper number)
paper/               LaTeX sources; build.py regenerates numbers.tex from JSON

Citing

Working paper: N. Angileri, Market States from Correlation Structure: Random-Matrix Spectra, Unsupervised Regimes, and Quantum Community Detection, MQFS Working Paper. Data: Kenneth R. French, Data Library, Tuck School of Business at Dartmouth. Key methods: Laloux et al. (PRL 1999); Mantegna (EPJ B 1999); Hamilton (Econometrica 1989); Farhi, Goldstone & Gutmann (arXiv:1411.4028); Hadfield et al. (Algorithms 2019); Havlíček et al. (Nature 2019); MacMahon & Garlaschelli (PRX 2015).

Development note

This project was built with AI-assisted engineering. Its architecture, the mathematics (Marchenko–Pastur cleaning, the minimum spanning tree, the HMM protocol, and the exact QUBO↔Ising mapping), and the validation were designed, reviewed, and are fully understood and owned by the author — the AI served as an implementation partner, not a substitute for judgement.

License

MIT — see LICENSE.

About

Market regimes from correlation structure: random-matrix spectra, correlation networks, out-of-sample HMM, and QAOA community detection

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages