feat(resilience): PR 0 — cohort-sanity release-gate harness#3369
Merged
Conversation
Lands the audit infrastructure for the resilience cohort-ranking structural audit (plan 2026-04-24-002). Release gate, not merge gate: the audit tells release review what to look at before publishing a ranking; it does not block a PR. What's new - scripts/audit-resilience-cohorts.mjs — Markdown report generator. Fetches the live ranking + per-country scores (or reads a fixture in offline mode), emits per-cohort per-dimension tables, contribution decomposition, saturated / outlier / identical-score flags, and a top-N movers comparison vs a baseline snapshot. - tests/resilience-construct-invariants.test.mts — 12 formula-level anchor-value assertions with synthetic inputs. Covers HHI, external debt (Greenspan-Guidotti anchor), and sovereign fiscal buffer (saturating transform). Tests the MATH, not a country's rank. - tests/fixtures/resilience-audit-fixture.json — offline fixture that mirrors the 2026-04-24 GCC state (KW>QA>AE) so the audit tool can be smoke-tested without API-key access. - docs/methodology/cohort-sanity-release-gate.md — operational doc explaining when to run, how to read the report, and the explicit anti-pattern note on rank-targeted acceptance criteria. Verified - `npx tsx --test tests/resilience-construct-invariants.test.mts` — 12 pass (HHI, debt, SWF invariants all green against current scorer) - `npm run test:data` — 6706 pass / 0 fail - `FIXTURE=tests/fixtures/resilience-audit-fixture.json OUT=/tmp/audit.md node scripts/audit-resilience-cohorts.mjs` runs to completion and correctly flags: (a) coverage-outlier on AE.importConcentration (0.3 vs peers 1.0) (b) saturated-high on GCC.externalDebtCoverage (all 6 at 100) — the two top cohort-sanity findings from the plan. Not in this PR - The live-API baseline snapshot (docs/snapshots/resilience-ranking-live-pre-cohort-audit-2026-04-24.json) is deferred to a manual release-prep step: run `WORLDMONITOR_API_KEY=wm_xxx API_BASE=https://api.worldmonitor.app node scripts/freeze-resilience-ranking.mjs` before the first methodology PR (PR 1 HHI period widening) so its movers table has something to compare against. - No scorer changes. No cache-prefix bumps. This PR is pure tooling.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Greptile SummaryThis PR ships the PR 0 audit harness for the cohort-sanity release gate: a Markdown report generator ( Confidence Score: 5/5Safe to merge — pure tooling/docs addition with no production code changes. All findings are P2: a misleading Source line in fixture-mode output, an unguarded file read, and a Unicode minus character. None affect correctness, production behaviour, or test reliability. The 12 invariant tests are well-scoped and the methodology doc is thorough. scripts/audit-resilience-cohorts.mjs — three minor polish items noted inline. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A([audit-resilience-cohorts.mjs]) --> B{FIXTURE set?}
B -- Yes --> C[Read fixture JSON
tests/fixtures/resilience-audit-fixture.json]
B -- No --> D[Fetch live ranking
GET /api/resilience/v1/get-resilience-ranking]
D --> E[fetchScoresConcurrent
GET /api/resilience/v1/get-resilience-score?cc=XX
concurrency=6]
C --> F[Build scoreMap]
E --> F
F --> G[decomposeContributions
per domain x dim x coverage x weight]
G --> H[flagDimensionPatterns
saturated-high / saturated-low
identical-scores / coverage-outlier]
H --> I{BASELINE set?}
I -- Yes --> J[computeMovers
top-N score deltas vs snapshot]
I -- No --> K[buildReport]
J --> K
K --> L{OUT set?}
L -- Yes --> M[Write OUT file]
L -- No --> N[stdout]
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+315
to
+326
| function computeMovers(currentItems, baselineItems, n) { | ||
| if (!baselineItems) return []; | ||
| const baselineByCc = new Map(baselineItems.map((x) => [x.countryCode, x])); | ||
| const currentByCc = new Map(currentItems.map((x) => [x.countryCode, x])); | ||
| const deltas = []; | ||
| for (const [cc, cur] of currentByCc.entries()) { | ||
| const prev = baselineByCc.get(cc); | ||
| if (!prev) continue; | ||
| const curScore = typeof cur.overallScore === 'number' ? cur.overallScore : null; | ||
| const prevScore = typeof prev.overallScoreRaw === 'number' ? prev.overallScoreRaw : (typeof prev.overallScore === 'number' ? prev.overallScore : null); | ||
| if (curScore == null || prevScore == null) continue; | ||
| deltas.push({ |
Contributor
There was a problem hiding this comment.
loadCountryNameMap crashes without useful diagnostics
If shared/country-names.json is missing or unparseable, the unhandled rejection surfaces as a raw stack trace with no actionable hint. A try/catch would produce a developer-friendly error and allow the script to continue with ISO-2 codes as fallback, which is especially important in CI-offline mode.
…la mode Addresses review P1 + P2 on PR #3369. P1 — fetch-failure silent-drop. Per-country score fetches that failed were logged to stderr, silently stored as null, and then filtered out of cohort tables via `codes.filter((cc) => scoreMap.get(cc))`. A transient 403/500 on the very country carrying the ranking anomaly could produce a Markdown report that looked valid — wrong failure mode for a release gate. Fix: - `fetchScoresConcurrent` now tracks failures in a dedicated Map and does NOT insert null placeholders; missing cohort members are computed against the requested cohort code set. - The report has a ⛔ blocker banner at top AND an always-rendered "Fetch failures / missing members" section (shown even when empty, so an operator learns to look). - `STRICT=1` writes the report, then exits code 3 on any fetch failure or missing cohort member, code 4 on formula-mode drift, code 0 otherwise. Automation can differentiate the two. P2 — pillar-combine formula mode invalidates contribution rows. `docs/methodology/cohort-sanity-release-gate.md:63` tells operators to run this audit before activating `RESILIENCE_PILLAR_COMBINE_ENABLED`, but the contribution decomposition is a domain-weighted roll-up that is ONLY valid when `overallScore = sum(domain.score * domain.weight)`. Once pillar combine is on, `overallScore = penalizedPillarScore(pillars)` (non-linear in dim scores); decomposition rows become materially misleading for exactly the release-gate scenario the doc prescribes. Fix: - Added `detectFormulaMode(scoreMap)` that takes countries with: (a) `sum(domain.weight)` within 0.05 of 1.0 (complete response), AND (b) every dim at `coverage ≥ 0.9` (stable share math) and compares `|Σ contributions - overallScore|` against `CONTRIB_TOLERANCE` (default 1.5). If > 50% of ≥ 3 eligible countries drift, pillar combine is flagged. - Report emits a ⛔ blocker banner at top, a "Formula mode" line in the header, and a "Formula-mode diagnostic" section with the first three offenders. Under `STRICT=1` exits code 4. - Methodology doc updated: new "Fail-closed semantics" section, "Formula mode" operator guide, ENV table entries for STRICT + CONTRIB_TOLERANCE. Verified: - `tests/audit-cohort-formula-detection.test.mts` (NEW) — 3 child-process smoke tests: missing-members banner + STRICT exit 3, all-clear exit 0, pillar-mode banner + STRICT exit 4. All pass. - `npx tsx --test tests/resilience-construct-invariants.test.mts tests/audit-cohort-formula-detection.test.mts` — 15 pass / 0 fail - `npm run test:data` — 6709 pass / 0 fail - `npm run typecheck` / `typecheck:api` — green - `npm run lint` / `lint:md` — no warnings on new / changed files (refactor split buildReport complexity from 51 → under 50 by extracting `renderCohortSection` + `renderDimCell`) - Fixture smoke: AE.importConcentration coverage-outlier and GCC.externalDebtCoverage saturated-high flags still fire correctly.
This was referenced Apr 24, 2026
Merged
Merged
Merged
…ountry-names, ASCII minus Addresses 3 P2 Greptile findings on #3369: 1. **Misleading Source: line in fixture mode.** `FIXTURE_PATH` sets `API_BASE=''`, so the report header showed a bare "/api/..." path that never resolved — making a fixture run visually indistinguishable from a live run. Now surfaces `Source: fixture://<path>` in fixture mode. 2. **`loadCountryNameMap` crashes without useful diagnostics.** A missing or unparseable `shared/country-names.json` produced a raw unhandled rejection. Now the read and the parse are each wrapped in their own try/catch; on either failure the script logs a developer-friendly warning and falls back to ISO-2 codes (report shows "AE" instead of "Uae"). Keeps the audit operable in CI-offline scenarios. 3. **Unicode minus `−` (U+2212) instead of ASCII `-` in `fmtDelta`.** Downstream operators diff / grep / CSV-pipe the report; the Unicode minus breaks byte-level text tooling. Replaced with ASCII hyphen- minus. Left the U+2212 in the formula-mode diagnostic prose (`|Σ contributions − overallScore|`) where it's mathematical notation, not data. Verified - `npx tsx --test tests/audit-cohort-formula-detection.test.mts tests/resilience-construct-invariants.test.mts` — 15 pass / 0 fail - Fixture-mode run produces `Source: fixture://tests/fixtures/...` - Movers-table negative deltas now use ASCII `-`
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…3372) PR 1 of cohort-audit plan 2026-04-24-002. Unblocks UAE, Oman, Bahrain (and any other late-reporter) on the importConcentration dimension. Problem - seed-recovery-import-hhi.mjs queries Comtrade with `period=Y-1,Y-2` (currently "2025,2024"). Several reporters publish Comtrade 1-2y behind — their 2024/2025 rows are empty while 2023 is populated. - With no data in the queried window, parseRecords() returned [] for the reporter, the seeder counted a "skip", the scorer fell through to IMPUTE (score=50, coverage=0.3, imputationClass="unmonitored"), and the cohort-sanity audit flagged AE as a coverage-outlier inside the GCC — exactly the class of silent gap the audit is designed to catch. Fix 1. Widen the Comtrade period parameter to a 4-year window Y-1..Y-4 via a new `buildPeriodParam(now)` helper. On-time reporters still pick their latest year via the existing completeness tiebreak in parseRecords(); late reporters now pick up whatever year they actually published in (2023 for UAE, etc.). 2. parseRecords() now returns { rows, year } — the year surfaces in the per-country payload as `year: number | null` for operator freshness audit. The scorer already expects this shape (_dimension-scorers.ts:1524 RecoveryImportHhiCountry.year); this PR actually populates it. 3. `buildPeriodParam` + `parseRecords` are exported so their unit tests can pin year-selection behaviour without hitting Comtrade. Note on PR 2 of the same plan The plan calls out "PR 2 — externalDebtCoverage re-goalpost to Greenspan-Guidotti" as unshipped. It IS shipped: commit 7f78a75 "PR 3 §3.5 point 3 — re-goalpost externalDebtCoverage (0..5 → 0..2)" landed under the prior workstream 2026-04-22-001. The new construct invariants in tests/resilience-construct-invariants.test.mts (shipped in PR 0 / #3369) confirm score(ratio=0)=100, score(1)=50, score(2)=0 against current main. PR 2 of the cohort-audit plan is a no-op; I'll flag this on the plan review thread rather than bundle a plan edit into this PR. Verified - `npx tsx --test tests/seed-recovery-import-hhi.test.mjs` — 19 pass (10 existing + 9 new: buildPeriodParam shape; parseRecords picks completeness-tiebreak, newer-year-on-ties, late-reporter fallback; empty/negative/world-aggregate handling) - `npx tsx --test tests/seed-comtrade-5xx-retry.test.mjs` — green (the `{ records, status }` destructure pattern at the caller still works; the new third field `year` is additive) - `npm run test:data` — 6703 pass / 0 fail - `npm run typecheck` / `typecheck:api` — green - `npm run lint` / `lint:md` — no new warnings - No cache-prefix bump: the payload shape only ADDS an optional field; old snapshots remain valid readers. Acceptance per plan - Construct invariant: score(HHI=0.05) > score(HHI=0.20) — already covered in tests/resilience-construct-invariants.test.mts (PR #3369) - Monotonicity pin: score(hhi=0.15) > score(hhi=0.45) — already covered in tests/resilience-dimension-monotonicity.test.mts Post-deploy verification After the next Railway seed-bundle-resilience-recovery cron tick, confirm UAE/OM/BH appear in `resilience:recovery:import-hhi:v1` with non-null hhi and `year` = 2023 (or their actual latest year). Then re-run the cohort audit — the GCC coverage-outlier flag on AE.importConcentration should disappear.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First PR in the cohort-audit workstream (plan:
docs/plans/2026-04-24-002-fix-resilience-cohort-ranking-structural-audit-plan.md). Ships only the release-gate audit harness — no scorer changes, no cache bumps.What landed
scripts/audit-resilience-cohorts.mjs— Markdown report generator. Fetches the live ranking + per-country scores (or reads a fixture in offline mode), emits per-cohort per-dimension tables, contribution decomposition, saturated / outlier / identical-score flags, and a top-N movers comparison vs a baseline snapshot.tests/resilience-construct-invariants.test.mts— 12 formula-level anchor-value assertions with synthetic inputs (not country identities). Tests the MATH, not a country's rank.HHI=0 → 100,HHI=0.5 → 0,score(HHI=0.05) > score(HHI=0.20)ratio=0 → 100,ratio=1.0 → 50(Greenspan-Guidotti),ratio=2.0 → 0em=0 → 0,em=12 → ≈63(1-yr saturating anchor),em=24 → ≈86tests/fixtures/resilience-audit-fixture.json— offline fixture mirroring the 2026-04-24 GCC state (KW>QA>AE) so the audit tool can be smoke-tested without API-key access.docs/methodology/cohort-sanity-release-gate.md— operational doc (when to run, how to read, explicit anti-pattern note on rank-targeted acceptance criteria).Testing
npx tsx --test tests/resilience-construct-invariants.test.mts→ 12 pass / 0 failnpm run test:data→ 6706 pass / 0 failnpm run typecheck/typecheck:api→ greennpm run lint/lint:md→ no errors on new filesFIXTURE=tests/fixtures/resilience-audit-fixture.json OUT=/tmp/audit.md node scripts/audit-resilience-cohorts.mjs→ runs to completion; correctly flags:These are exactly the two top cohort-sanity findings the plan identifies.
What this PR explicitly does NOT do
cohort-ranking-sanity-surfaces-hidden-data-gaps.Pre-release manual step (deferred from this PR)
Before the first methodology PR (PR 1 HHI period widening) merges, freeze the live baseline snapshot so the audit's movers table has a reference:
Deferred because the resilience endpoints are in
PREMIUM_RPC_PATHSand this execution environment has no API key.Post-Deploy Monitoring & Validation
No additional operational monitoring required: this PR adds a diagnostic script, a test file, a fixture, and a doc. Nothing runs in production; no endpoints, crons, or flags are touched.
Related
docs/plans/2026-04-24-002-fix-resilience-cohort-ranking-structural-audit-plan.mddocs/plans/2026-04-22-001-fix-resilience-scorer-structural-bias-plan.mdcohort-ranking-sanity-surfaces-hidden-data-gaps🤖 Generated with Claude Opus 4.7 (1M context) via Claude Code + Compound Engineering v2.49.0
Co-Authored-By: Claude Opus 4.7 (1M context) [email protected]