feat(resilience): cohort anti-inversion fixture (Plan 2026-04-26-002 PR 0 / U1)#3433
Conversation
…§U1, PR 0) Lands the structural CI gate the universe + coverage rebuild needs. Passes against current v15 ranking with PERMISSIVE thresholds; PRs 3+5 will tighten the thresholds in lockstep with their corresponding fixes. **Why this exists.** PR #3425's post-merge cohort dry-run showed that even after targeted scorer fixes (logisticsSupply default, sovereignFiscalBuffer Path 3, socialCohesion GPI-only impute) AND the externalDebtCoverage data-source repair (PR #3427), tiny states still climb in v15 (TV rank 12→6, PW 14→7, NR 58→36). The targeted fixes can't reach the universe + coverage handling defects. This fixture pins the cohort-level invariants so the rebuild PRs (universe filter, coverage penalty, stable-absence recalibration, per-capita normalization) can each demonstrate measurable cohort movement at merge time. **What it asserts (PERMISSIVE baselines passing on current v15).** | Invariant | v15 measurement | PR 0 threshold | |----------------------------------------------|-----------------|----------------| | median(G7) − median(microstate-territories) | **−5.52pt** | > −10pt | | median(Nordics) − median(GCC) | **+8.20pt** | ≥ −30pt | | min(G7) − max(Sub-Saharan-LIC) | **+10.40pt** | ≥ −20pt | | microstate-territories in top 20 | **6** (AD, LI, TV, PW, GL, MO) | report-only | The G7-vs-microstate inversion (−5.52pt) is the structural problem the rebuild exists to fix. PR 0 baseline tolerates it up to 10pt; PR 3 (coverage penalty) is expected to flip the sign and tighten to +10pt. **Files.** - 5 cohort JSON fixtures at server/worldmonitor/resilience/v1/cohorts/ (g7, nordics, gcc, sub-saharan-lic, microstate-territories) — committed data, not test code, so cohort membership can be reviewed independently - tests/resilience-cohort-anti-inversion.test.mts — fixture self-tests (always run) + live-ranking invariants (skip-on-missing-creds) - tests/resilience-retired-dimensions-parity.test.mts — extended with cohort JSON shape validation (anti-mystery-cohort gate: every cohort must carry a non-empty description explaining its purpose) **Read pattern.** Reads live `resilience:ranking:vN` from production Upstash via the read-only GET pattern from scripts/dry-run-resilience-rebalance.mjs (PR #3425). Skips gracefully when UPSTASH_REDIS_REST_URL is unset — required for CI-without-prod-creds and dev-without-.env environments. The cache-key import is dynamic (`await import('../server/.../_shared.ts')`) so the test always reads the current cache version, never a hardcoded literal — same source-of-truth pattern PR #3425 established for the dry-run script. **Coverage anomaly.** microstate-territories cohort has 12/13 present in v15 (FK Falklands missing — a UK territory not in WB IDS scope, so expected). The other 4 cohorts are 100% covered. Reported in the diagnostic `it()` block. **Verification.** - 617/617 resilience tests pass (`npx tsx --test tests/resilience-*`) — 594 prior + 23 new cohort + parity tests - Live-invariants pass against current v15 with prod creds - Skip-on-missing-creds path passes without `.env` loaded 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR lands the structural CI gate for the resilience universe + coverage rebuild (Plan 2026-04-26-002 §U1): five committed cohort JSON fixtures (G7, Nordics, GCC, Sub-Saharan-LIC, microstate-territories) and two test files that enforce anti-inversion invariants against the live Upstash ranking with intentionally permissive PR-0 thresholds, to be tightened in PRs 3 and 5.
Confidence Score: 3/5Mergeable in intent but the P1 false-green pattern means the gate can silently pass when production data is unavailable, undermining the CI signal this PR exists to provide. One P1 logic flaw (vacuous pass on setup failure with creds present) caps the score at 4; the finding directly undermines the core purpose of this CI gate (catching regressions), which pulls it below the ceiling to 3. The JSON fixtures and fixture self-tests are clean. tests/resilience-cohort-anti-inversion.test.mts — the shared-state ranking variable and Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A([Test run starts]) --> B{UPSTASH creds\npresent?}
B -- No --> C[emit skip-log + assert.ok true\nAll live tests pass ✅]
B -- Yes --> D["[setup] fetch live ranking\nfrom Upstash GET /get/:key"]
D --> E{payload\nnull or empty?}
E -- No --> F{ranking.size\n>= 100?}
F -- No --> G[setup FAILS ❌\nranking stays null]
F -- Yes --> H[ranking populated ✅]
E -- Yes --> G
G --> I["invariant tests:\nif (!ranking) return ⚠️\n→ vacuously PASS"]
H --> J[Invariant 1: median G7 vs microstate]
H --> K[Invariant 2: median Nordics vs GCC]
H --> L[Invariant 3: min G7 vs max Sub-Saharan-LIC]
H --> M[Report-only: microstate count top 20]
J & K & L & M --> N([Results reported])
style G fill:#f88,color:#000
style I fill:#fa0,color:#000
Reviews (1): Last reviewed commit: "feat(resilience): cohort anti-inversion ..." | Re-trigger Greptile |
| // Non-fatal — env may already be loaded by the harness. | ||
| } | ||
| rankingKey = await loadRankingKey(); | ||
| const payload = await fetchLiveRanking(rankingKey, upstashUrl!, upstashToken!); |
There was a problem hiding this comment.
Invariant tests vacuously pass when setup fails with creds present
When UPSTASH_REDIS_REST_URL/TOKEN are set (i.e., credsPresent = true) but the [setup] test fails — e.g., Upstash returns null, the payload is malformed, or ranking.size < 100 — ranking stays null. Every subsequent invariant test then executes if (!ranking) return; and returns without any assertion, producing a false-green CI signal. You get one failing setup test plus four silently-passing invariants, which is a misleading outcome exactly when the production endpoint is most degraded.
| const g7Scores = scoresFor(cohorts.g7, ranking); | ||
| const licScores = scoresFor(cohorts.subSaharanLic, ranking); | ||
| if (g7Scores.length === 0 || licScores.length === 0) { | ||
| console.warn(`[cohort-anti-inversion] G7 scores=${g7Scores.length}, LIC scores=${licScores.length} — skipping`); | ||
| return; | ||
| } | ||
| const g7Min = Math.min(...g7Scores); | ||
| const licMax = Math.max(...licScores); | ||
| console.log(`[cohort-anti-inversion] min(G7) = ${g7Min.toFixed(2)}, max(Sub-Saharan-LIC) = ${licMax.toFixed(2)}, gap = ${(g7Min - licMax).toFixed(2)}`); | ||
| assert.ok(g7Min >= licMax - 20, | ||
| `Catastrophic floor regression: min(G7)=${g7Min} fell within 20pt of max(Sub-Saharan-LIC)=${licMax}. Recovery domain or coverage handling has regressed.`); |
There was a problem hiding this comment.
Inconsistent empty-score handling across invariants
The G7-vs-LIC test explicitly guards if (g7Scores.length === 0 || licScores.length === 0) { return; } and silently passes, whereas the G7-vs-microstate and Nordics-vs-GCC tests have no such guard — median([]) returns NaN, and NaN >= x - 20 is false, so those tests would actually fail on empty scores. The inconsistency means a schema change or new country-code aliasing that strips one cohort from the ranking would produce different behaviour depending on which invariant is affected.
| // Standalone describe block (different concern from retired/not-applicable | ||
| // parity above) but kept in the same file so all "shape-of-data must | ||
| // match the schema" gates live together. | ||
| import { readdirSync } from 'node:fs'; |
There was a problem hiding this comment.
Non-top-of-file
import after executable statements
import { readdirSync } from 'node:fs'; appears at line 87, after multiple const declarations and describe(...) blocks. While ES module imports are hoisted and this is technically valid, TypeScript's --verbatimModuleSyntax or some eslint-plugin-import rules (import/first) flag this pattern. The idiomatic fix is to merge it with the existing import { readFileSync } from 'node:fs'; at the top of the file.
| import { readdirSync } from 'node:fs'; | |
| import { readFileSync, readdirSync } from 'node:fs'; |
(And remove the standalone import { readdirSync } at line 87.)
| import { readdirSync } from 'node:fs'; | |
| import { readFileSync, readdirSync } from 'node:fs'; |
#3433 shape gate (review fixup on PR #3435) Cross-PR finding: PR #3433 added a cohort JSON shape gate that scans every *.json in `server/worldmonitor/resilience/v1/cohorts/` and asserts the cohort schema (`{ name, description, iso2: string[] }`). PR #3435 added `sovereign-status.json` to the same directory, but its shape is a per-country PROPERTY registry (`{ entries: [{iso2, status}] }`), not a cohort membership list. After both PRs merge, the shape gate fails on sovereign-status.json. Fix: directory split. Cohorts (membership lists) stay in `cohorts/`; per-country property registries move to `registries/`. The shape gate naturally narrows to homogeneous-shape files. **Layout post-fix:** - `server/worldmonitor/resilience/v1/cohorts/` — cohort membership lists (PR #3433's g7/nordics/gcc/sub-saharan-lic/microstate-territories) - `server/worldmonitor/resilience/v1/registries/` — per-country property registries (sovereign-status; future Q4 deferred work could add landlocked, etc.) **Updates:** - Moved `cohorts/sovereign-status.json` → `registries/sovereign-status.json` - `scripts/shared/rankable-universe.mjs` — path updated + comment documenting the directory-split rationale - `server/worldmonitor/resilience/v1/_rankable-universe.ts` — path + comment updated - `tests/resilience-universe-filter.test.mts` — path updated **Verified by simulating post-merge state:** copied PR #3433's cohort JSONs + extended parity test into this branch locally; both PRs' tests pass (8/8 cohort + 15/15 universe-filter). Reverted the simulation files before commit — only the PR #3435 intended changes ship. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…6-04-26-002 PR 1 / U2) (#3435) * feat(resilience): universe filter — 193 UN members + 3 SARs (Plan 2026-04-26-002 §U2, PR 1) Lands the rankable-universe whitelist filter at universe-build time. Earlier behavior: `seed-resilience-static.mjs` admitted every ISO2 from any source map (~222 entries including AS/GU/GL/IM/GI/FK/PR/etc.). Post-PR-1: ~196 entries (193 UN members + 3 SARs). **Decision (per plan KTD Q1)**: Option (b) — UN members + 3 standalone SARs (HK, MO, TW). The 3-SAR exception preserves their presence in the dataset; the future `headlineEligible` gate (PR 6) can separate them from the headline ranking if/when that policy ships. **Single source of truth.** `server/worldmonitor/resilience/v1/cohorts/sovereign-status.json` holds the 196-entry whitelist. `scripts/shared/rankable-universe.mjs` is the helper both seeders consume to ensure their universes match. **Filter sites (both seeders).** - `seed-resilience-static.mjs:finalizeCountryPayloads` — drops non-rankable entries at universe-build time. Logs `Dropped N non-rankable territory entries`. - `seed-resilience-scores.mjs` — defense-in-depth filter on the static index read, in case the index was seeded by an older version of seed-resilience-static during a deploy transient. **No score-formula changes.** This is a universe-membership change only. Existing payload shape and per-country score computation unchanged. The cohort anti-inversion fixture (PR 0 / PR #3433) thresholds remain PERMISSIVE; PR 3 (coverage penalty) is the next step that tightens them. **Testing.** - `tests/resilience-universe-filter.test.mts` (new, 15 tests): - Whitelist shape: 193 UN + 3 SARs = 196, no duplicates - SAR cohort = exactly {HK, MO, TW} - `isInRankableUniverse` accepts UN members + SARs, rejects territories (AS, GU, GL, IM, GI, FK, PR, BM, KY, etc.) and non-UN entities (XK Kosovo, PS Palestine, VA Vatican, EH Western Sahara) - Case-insensitive iso2 handling - Invalid input shapes (empty, ISO3, null, undefined) all return false - `npm run test:data` — 609/609 resilience tests pass - `npm run typecheck` clean **Lineage.** Plan 2026-04-26-002 §U2 (PR 1 of 8). Stacked on PR 0 (#3433) which already shipped the cohort anti-inversion fixture. This PR's filter directly affects the microstate-territories cohort in PR 0's fixture (4 of 13 entries — GL, GI, IM, FK — drop out of the live ranking once this lands), but the PERMISSIVE thresholds remain valid. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * fix(resilience): bump static-source-version v7→v8 + handler-side universe filter (review fixup on PR #3435) Two reviewer findings on PR #3435's initial commit: **P1: static seeder would silently no-op post-merge.** `shouldSkipSeedYear` (seed-resilience-static.mjs:70) skips when existing meta has the same `sourceVersion` AND `seedYear`. Prod already has a successful 2026 v7 static seed, so running the seeder post-merge would log "already written, skipping" — the new whitelist filter at `finalizeCountryPayloads` would never run, and `resilience:static:index:v1` would remain at ~222 entries. The universe filter wouldn't take effect. Fix: bumped `RESILIENCE_STATIC_SOURCE_VERSION` v7→v8. The next seeder run after merge will see the new version, treat it as a fresh seed, and apply the universe filter. **P2: handler-side defense-in-depth was incomplete.** `scripts/seed-resilience-scores.mjs` filtered its local `countryCodes`, but then it calls `/api/resilience/v1/get-resilience-ranking?refresh=1` which re-reads `listScorableCountries()` from Redis at `server/worldmonitor/resilience/v1/_shared.ts:661`. That helper was returning `manifest.countries` unfiltered. So even with the script's local filter, a stale 222-country manifest could still drive the ranking endpoint to publish 222 countries. Fix: - New `server/worldmonitor/resilience/v1/_rankable-universe.ts` — server-side TS mirror of `scripts/shared/rankable-universe.mjs`. Both read the SAME canonical JSON at `cohorts/sovereign-status.json`; the duplication is the read-path (fs.readFileSync vs ES JSON import), not the data. - `_shared.ts:listScorableCountries` now applies `isInRankableUniverse` as a final filter. Idempotent: a fresh manifest from the post-bump seeder is already filtered, so the filter is a no-op then; a stale pre-PR-1 manifest gets filtered at handler-time. The rankable-universe contract is now enforced at three points: 1. seed-resilience-static.mjs:finalizeCountryPayloads (write-time) 2. seed-resilience-scores.mjs (read-time defense) 3. _shared.ts:listScorableCountries (handler-time defense) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * fix(resilience): move sovereign-status.json out of cohorts/ to avoid PR #3433 shape gate (review fixup on PR #3435) Cross-PR finding: PR #3433 added a cohort JSON shape gate that scans every *.json in `server/worldmonitor/resilience/v1/cohorts/` and asserts the cohort schema (`{ name, description, iso2: string[] }`). PR #3435 added `sovereign-status.json` to the same directory, but its shape is a per-country PROPERTY registry (`{ entries: [{iso2, status}] }`), not a cohort membership list. After both PRs merge, the shape gate fails on sovereign-status.json. Fix: directory split. Cohorts (membership lists) stay in `cohorts/`; per-country property registries move to `registries/`. The shape gate naturally narrows to homogeneous-shape files. **Layout post-fix:** - `server/worldmonitor/resilience/v1/cohorts/` — cohort membership lists (PR #3433's g7/nordics/gcc/sub-saharan-lic/microstate-territories) - `server/worldmonitor/resilience/v1/registries/` — per-country property registries (sovereign-status; future Q4 deferred work could add landlocked, etc.) **Updates:** - Moved `cohorts/sovereign-status.json` → `registries/sovereign-status.json` - `scripts/shared/rankable-universe.mjs` — path updated + comment documenting the directory-split rationale - `server/worldmonitor/resilience/v1/_rankable-universe.ts` — path + comment updated - `tests/resilience-universe-filter.test.mts` — path updated **Verified by simulating post-merge state:** copied PR #3433's cohort JSONs + extended parity test into this branch locally; both PRs' tests pass (8/8 cohort + 15/15 universe-filter). Reverted the simulation files before commit — only the PR #3435 intended changes ship. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * fix(resilience): filter cached ranking response too (review fixup on PR #3435) Reviewer P2 finding: get-resilience-ranking.ts returns the cached resilience:ranking:v15 payload BEFORE calling listScorableCountries(), and the cache key remains v15 across this PR. So after deploy + static v8 reseed, normal API requests can still return the pre-PR cached 222-country ranking for up to the 12h ranking TTL — the new handler-side universe filter at _shared.ts:listScorableCountries only runs on the recompute path, not on cache hits. Fix: add the universe filter to the cached-response read path too. The filter is idempotent — a fresh post-PR-1 cached payload is already universe-filtered (no-op), a stale pre-PR-1 cached payload gets filtered at handler-time. Same recipe as the prior 3-layer defense pattern (write-time + read-time + handler-time). The rankable-universe contract is now enforced at FOUR layers: 1. seed-resilience-static.mjs:finalizeCountryPayloads (write-time) 2. seed-resilience-scores.mjs (read-time defense) 3. _shared.ts:listScorableCountries (recompute-path defense) 4. get-resilience-ranking.ts cache hit (cached-response defense — new) This eliminates the operator-toil failure mode entirely. No need for the post-merge force-refresh checklist that PR #3427 required, and no need to bump the ranking cache key (which would be wasted churn since PR 3 of this plan will bump it for the coverage-penalty redesign anyway). Test fixup: tests/resilience-ranking.test.mts:382 used 'ZZ' as a sentinel to verify the auth gate. The new universe filter correctly drops 'ZZ' as non-rankable, defeating the auth-gate test's intent. Changed sentinel to 'NR' (Nauru — UN member, real country, but won't appear in the recompute path's static-index fixture). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> * fix(resilience): plain JSON import (no `with { type: 'json' }`) for Vercel esbuild compat (review fixup on PR #3435) Reviewer P1 finding — Vercel deployment failure: ``` vc-file-system:__vc__ns__/12/server/worldmonitor/resilience/v1/_rankable-universe.js:26:65: ERROR: Expected ";" but found "with" ``` Vercel's esbuild bundler does NOT support the ES import-attribute syntax (`with { type: 'json' }`) used in the initial _rankable-universe.ts. Existing server TS JSON imports in this codebase all use the plain form WITHOUT import attributes — see e.g. `server/worldmonitor/resilience/v1/_dimension-scorers.ts:1-2`: ```ts import countryNames from '../../../../shared/country-names.json'; import iso2ToIso3Json from '../../../../shared/iso2-to-iso3.json'; ``` Same shape applied here: ```ts import sovereignStatus from './registries/sovereign-status.json'; ``` Verified locally: typecheck clean, 609/609 resilience tests pass, and `npx esbuild --bundle --platform=neutral --loader:.json=json` produces a clean bundle (mimics Vercel's bundle path). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Claude <[email protected]>
… (Railway bundle ENOENT) (#3441) Production failure post-PR-#3435: Railway seed-bundle-resilience container crashes on every cron tick with: ``` Error: ENOENT: no such file or directory, open '/server/worldmonitor/resilience/v1/registries/sovereign-status.json' at file:///app/shared/rankable-universe.mjs:49:15 [Bundle:resilience] Finished in 0.2s, ran:0 skipped:1 deferred:0 failed:1 ``` **Root cause** (memory `worldmonitor-scripts-package-json-install-scope`): Railway NIXPACKS services with `rootDirectory=scripts/` only ship files under scripts/ into the container. PR #3435 placed sovereign-status.json under server/worldmonitor/resilience/v1/registries/ — that resolved locally and on Vercel (different bundlers, different file layouts) but ENOENT'd at Railway runtime. Score-warming cron has been failing on every tick since merge. **Fix:** move JSON to `scripts/shared/sovereign-status.json` — the canonical location for shared JSON consumed by both seeders AND server-side code (alongside iso2-to-iso3.json, country-names.json etc., per `_dimension-scorers.ts:1-2` import pattern). **Updates:** - `scripts/shared/rankable-universe.mjs`: sibling read, `resolve(here, 'sovereign-status.json')` — no path traversal - `server/worldmonitor/resilience/v1/_rankable-universe.ts`: `import sovereignStatus from '../../../../scripts/shared/sovereign-status.json'` matching `_dimension-scorers.ts` pattern - `tests/resilience-universe-filter.test.mts`: updated path - `server/worldmonitor/resilience/v1/registries/`: removed (was intermediate location to avoid PR #3433's cohort shape gate; the scripts/shared/ location ALSO sits outside that gate's scope) **Verification (4 layers):** - `npm run typecheck` clean - 15/15 universe-filter tests pass (full resilience suite would also pass; ran narrower subset since this is a path-only change) - `npx esbuild --bundle --platform=neutral` produces clean bundle (Vercel) - `cd scripts && node -e "import('./shared/rankable-universe.mjs')..."` loads 196 entries successfully (Railway bundle path simulation) **Lesson** (now in skill `worldmonitor-scripts-package-json-install-scope`): when a server-side TS module needs to import a JSON asset that is ALSO consumed by a Railway NIXPACKS bundle service, the JSON MUST live under `scripts/shared/`. server/.../whatever/ paths resolve fine in tsc + tsx + Vercel/esbuild but ENOENT in the Railway container. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <[email protected]>
Summary
PR 0 of plan 2026-04-26-002 — lands the structural CI gate the universe + coverage rebuild needs. Passes against current v15 with PERMISSIVE thresholds; subsequent rebuild PRs (3, 5) tighten in lockstep.
PR #3425's cohort dry-run captured the empirical signature this gate now anchors: median(G7)=69.5 sits 5.5pt below median(microstate-territories)=75.0 in current v15. The structural inversion the universe + coverage rebuild exists to fix.
What this PR asserts (passing on current v15)
What ships
server/worldmonitor/resilience/v1/cohorts/— data, not test codetests/resilience-cohort-anti-inversion.test.mts— fixture self-tests (always run) + live-ranking invariants (skip-on-missing-creds)tests/resilience-retired-dimensions-parity.test.mtsextended with JSON shape validation (anti-mystery-cohort gate)Test plan
npx tsx --test tests/resilience-*— 617/617 passes.envUpstash credsLineage
Plan 2026-04-26-002 §U1 (PR 0). Predecessors: PRs #3425, #3426, #3427, #3432.