feat(resilience): external index correlation validation (ND-GAIN, INFORM)#2824
Conversation
…ORM) Batch validation script computing Spearman rho between WorldMonitor resilience scores and ND-GAIN readiness / INFORM risk indices for 50 representative countries. Identifies top divergences. Phase 4 gate: rho > 0.6 with at least 2 benchmark indices.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR adds Confidence Score: 5/5Safe to merge — script is a read-only diagnostic tool with no changes to production scoring code. All findings are P2: one piece of dead code ( No files require special attention.
|
| Filename | Overview |
|---|---|
| scripts/validate-resilience-correlation.mjs | New read-only diagnostic script: fetches Redis resilience scores and computes Spearman rho against hardcoded ND-GAIN and INFORM reference data. Core statistics (toRanks, pearson, spearmanRho) are implemented correctly with proper tied-rank handling. Minor issues: redisGetJson helper is defined but never called (dead code), and the gate check implements 2-of-2 (both must pass) rather than the stated 2-of-3 with the ND-GAIN threshold set at > 0.65 instead of the described uniform > 0.60. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[run] --> B[loadEnvFile / getRedisCredentials]
B --> C[fetchWorldMonitorScores\nredisPipeline GET resilience:score:v4:ISO2]
C --> D{wmScores.size < 20?}
D -- yes --> E[process.exit 1]
D -- no --> F[computeCorrelation\nvs ND-GAIN invert=false]
D -- no --> G[computeCorrelation\nvs INFORM invert=true]
F --> H[spearmanRho\ntoRanks + pearson on ranks]
G --> H
H --> I[Rank divergences\nsorted by delta]
F --> J{ndgainPass\nrho > 0.65?}
G --> K{informPass\nrho > 0.60?}
J --> L[passingCount]
K --> L
L --> M{passingCount >= 2?}
M -- YES --> N[GATE PASS — print report]
M -- NO --> O[GATE FAIL — print report]
Reviews (1): Last reviewed commit: "feat(resilience): external index correla..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| async function redisGetJson(url, token, key) { | ||
| const resp = await fetch(`${url}/get/${encodeURIComponent(key)}`, { | ||
| headers: { Authorization: `Bearer ${token}` }, | ||
| signal: AbortSignal.timeout(5_000), | ||
| }); | ||
| if (!resp.ok) return null; | ||
| const data = await resp.json(); | ||
| if (!data?.result) return null; | ||
| try { return JSON.parse(data.result); } catch { return null; } | ||
| } |
There was a problem hiding this comment.
redisGetJson is never called
redisGetJson is defined but not used anywhere in the file. All score fetching is done through redisPipeline inside fetchWorldMonitorScores (line 91). Since the exported API surface only exposes the statistical functions and computeCorrelation, this helper won't be reached from import callers either. Consider removing it to avoid confusion.
| const ndgainPass = ndgainResult.rho > 0.65; | ||
| const informPass = informResult.rho > 0.60; | ||
|
|
||
| console.log(`WorldMonitor vs ND-GAIN Readiness: rho = ${ndgainResult.rho.toFixed(3)} (n=${ndgainResult.n}, target > 0.65) ${ndgainPass ? 'PASS' : 'FAIL'}`); | ||
| console.log(`WorldMonitor vs INFORM Risk: rho = ${informResult.rho.toFixed(3)} (n=${informResult.n}, target > 0.60, inverted) ${informPass ? 'PASS' : 'FAIL'}`); | ||
|
|
||
| const passingCount = [ndgainPass, informPass].filter(Boolean).length; | ||
| const gatePass = passingCount >= 2; | ||
| console.log(`\nGATE CHECK: rho > 0.6 for at least 2 indices? ${gatePass ? 'YES' : 'NO'} (${passingCount}/2 passing)\n`); |
There was a problem hiding this comment.
The PR description states the gate is "rho > 0.6 with at least 2 of 3 benchmark indices," but only two indices are implemented here. The gate condition passingCount >= 2 with exactly two indices is equivalent to requiring both to pass — which is stricter than "2 of 3." Additionally, the ND-GAIN threshold is > 0.65 (line 162) while the described gate threshold is > 0.60 uniformly; the INFORM threshold matches at > 0.60. If a third index is planned, the gate logic will need updating when it is added. As written, the GATE CHECK console message saying (${passingCount}/2 passing) correctly reports what is implemented, but the mismatch with the PR description may cause confusion during a phase review.
Was hardcoded to v4, but production is v5 after PR #2821 (baseline/stress engine). Script would miss all cached scores and fail with "Too few scores available".
1 participant
Summary
scripts/validate-resilience-correlation.mjscomputing Spearman rank correlation (rho) between WorldMonitor resilience scores and two established benchmark indices: ND-GAIN Readiness and INFORM Risk IndexPhase 4 gate: rho > 0.6 with at least 2 of 3 benchmark indices.
Implementation details
resilience:score:v4:<ISO2>keys from Redis (same asseed-resilience-scores.mjs)spearmanRho,toRanks,pearson,computeCorrelation) for future test useTest plan
node --check)toRanks([10, 20, 20, 30])produces[1, 2.5, 2.5, 4]