docs(resilience): PR 5.2 — displacement field-mapping audit + known-limitations#3373
Merged
Conversation
…imitations
PR 5.2 of cohort-audit plan 2026-04-24-002. Read-only static audit of
the UNHCR displacement field mapping consumed by scoreSocialCohesion,
scoreBorderSecurity, and scoreStateContinuity.
Findings
1. Field mapping is CODE-CORRECT. The plan's concern — that
`totalDisplaced` might inadvertently include labor migrants — is
negative at the source. The UNHCR Population API does not publish
labor migrant data at all; it covers only four categories
(refugees, asylum seekers, IDPs, stateless), all of which the
seeder sums correctly. Labor-migrant-dominated cohorts (GCC, SG)
legitimately register as "no UNHCR footprint" — that's UNHCR
semantics, not a bug.
2. NEW finding during audit — `scoreBorderSecurity` fallback at
_dimension-scorers.ts:1412 is effectively dead code. The
`hostTotal ?? totalDisplaced` fallback never fires in production
for two compounding reasons:
(a) `safeNum(null)` returns 0 (JS `Number(null) === 0`), so the
`??` short-circuits on 0 — the nullish-coalescing only falls
back on null/undefined.
(b) `scripts/seed-displacement-summary.mjs` ALWAYS writes
`hostTotal: 0` explicitly for origin-only countries (lines
141-144). There's no production shape where `hostTotal` is
undefined, so the `??` can never select the fallback path.
Observable consequence: origin-only high-outflow countries
(Syria, Venezuela, Ukraine, Afghanistan) score 100 on
borderSecurity's displacement sub-component (35% of the dim
blend). The outflow signal is effectively silenced.
3. NOT fixing this in this PR. A one-line change (`||` or an
explicit `> 0` check) would flip the borderSecurity score for
~6 high-outflow origin countries by a material amount — a
methodology change, not a pure bug-fix. Belongs in a construct-
decision PR with before/after cohort snapshots. Opening this as
a follow-up discussion instead of bundling into an audit-doc PR.
Shipped
- `docs/methodology/known-limitations.md` — new file. Sections:
"Displacement field-mapping" covering source semantics (what
UNHCR provides vs does not), the GCC labor-migrant-cohort
implication, the `??` short-circuit finding, and the decision
to not fix in this PR. Includes a follow-up audit list of 11
countries (high host-pressure + high origin-outflow + labor-
migrant cohorts) for a live-data spot-check against UNHCR
Refugee Data Finder — gated on API-key access.
- `tests/resilience-displacement-field-mapping.test.mts` —
9-test regression guard. Pins:
(1) `totalDisplaced` = sum of all four UNHCR categories;
(2) `hostTotal` = asylum-side sum (no IDPs/stateless);
(3) stateless population flows into totalDisplaced (guards
against a future seeder refactor that drops the term);
(4) labor-migrant-cohort (UNHCR-empty) entry scores 100 on
the displacement sub-component — the correct-per-UNHCR-
semantics outcome, intentionally preserved;
(5) CURRENT scoreBorderSecurity behaviour: hostTotal=0
short-circuits `??` (Syria-pattern scores 100);
(6) `??` fallback ONLY fires when hostTotal is undefined
(academic; seeder never emits this shape today);
(7) `safeNum(null)` returns 0 quirk pinned as a numeric-
coercion contract;
(8) absent-from-UNHCR country imputes `stable-absence`;
(9) scoreStateContinuity reads `totalDisplaced` origin-side.
Verified
- `npx tsx --test tests/resilience-displacement-field-mapping.test.mts` — 9 pass / 0 fail
- `npm run test:data` — 6703 pass / 0 fail
- `npm run typecheck` / `typecheck:api` — green
- `npm run lint` / `lint:md` — no warnings on new files
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile SummaryThis is a read-only audit PR adding a methodology doc and 9 regression-guard tests for the UNHCR displacement field mapping consumed by Confidence Score: 5/5Safe to merge — pure docs + regression-guard tests with zero scorer behaviour changes. All findings are P2 or lower. The only code issue (truncated assertion message on line 169) does not affect test correctness or production behaviour, only failure diagnostics. tests/resilience-displacement-field-mapping.test.mts line 169 — minor assertion-message truncation worth fixing. Important Files Changed
Reviews (1): Last reviewed commit: "docs(resilience): PR 5.2 — displacement ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ]); | ||
| const origin = await scoreBorderSecurity('SYR', makeReader(originPayload)); | ||
| assert.equal(origin.score, 100, | ||
| `origin-only country scores 100 today (the `??` fallback does NOT fire on hostTotal=0); got ${origin.score}`); |
Contributor
There was a problem hiding this comment.
The backtick characters wrapping ?? in this template literal are unescaped. JavaScript parses this as string1 ?? string2 — the ?? operator returns string1 (a non-null string), so the assert.equal message argument receives only "origin-only country scores 100 today (the ". If the assertion fails, the message is silently truncated: the description of the modeling note and the got ${origin.score} diagnostic are both swallowed. The tests still run and pass correctly, but a regression would surface with a confusing, incomplete failure message.
Suggested change
| `origin-only country scores 100 today (the `??` fallback does NOT fire on hostTotal=0); got ${origin.score}`); | |
| `origin-only country scores 100 today (the \`??\` fallback does NOT fire on hostTotal=0); got ${origin.score}`); |
This was referenced Apr 24, 2026
Merged
Merged
Merged
Addresses Greptile P2 on #3373. The unescaped backticks around the nullish-coalescing operator in a template literal caused JavaScript to parse the string as 'prefix' ?? 'suffix' — truncating the assertion message to the prefix alone on failure. Escaping the backticks preserves the full diagnostic so a future regression shows the complete context. Semantics unchanged; test still passes.
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…istic GCC identity) PR 5.3 of cohort-audit plan 2026-04-24-002. Stacked on PR 5.2 (#3373) so the known-limitations.md section append is additive. Read-only static audit of scoreFoodWater. Findings 1. The observed GCC-all-score-53 is CONSTRUCT-DETERMINISTIC, not a regional-default leak. Pinned mathematically: - IPC/HDX doesn't publish active food-crisis data for food-secure states → scorer's fao-null branch imputes IMPUTE.ipcFood=88 (class='stable-absence', cov=0.7) at combined weight 0.6 - WB indicator ER.H2O.FWST.ZS (labelled 'water stress') for GCC is EXTREME (KW ~3200%, BH ~3400%, UAE ~2080%, QA ~770%) — all clamp to sub-score 0 under the scorer's lower-better 0..100 normaliser at weight 0.4 - Blended with peopleInCrisis=0 (fao block present with zero): (100 * 0.45 + 0 * 0.4) / (0.45 + 0.4) = 45 / 0.85 ≈ 53 Every GCC country has the same inputs → same outputs. That's construct math, not a regional lookup. 2. Indicator-keyword routing is code-correct. `'water stress'`, `'withdrawal'`, `'dependency'` route to lower-better; `'availability'`, `'renewable'`, `'access'` route to higher-better; unrecognized indicators fall through to a value-range heuristic with a WARN log. 3. No bug or methodology decision required. The 53-all-GCC output is a correct summary statement: "non-crisis food security + severe water-withdrawal stress." A future construct decision might split foodWater into separate food and water dims so one saturated sub-signal doesn't dominate the combined dim for desert economies — but that's a construct redesign, not a bug. Shipped - `docs/methodology/known-limitations.md` — extended with a new section documenting the foodWater audit findings, the exact blend math that yields ~53 for GCC, cohort-determinism vs regional-default, and a follow-up data-side spot-check list gated on API-key access. - `tests/resilience-foodwater-field-mapping.test.mts` — 8 new regression-guard tests: 1. indicator='water stress' routes to lower-better 2. GCC extreme-withdrawal anchor (value=2000 → blended score 53) 3. indicator='renewable water availability' routes to higher-better 4. fao=null with static record → imputes 88; imputationClass=null because observed AQUASTAT wins (weightedBlend T1.7 rule) 5. fully-imputed (fao=null + aquastat=null) surfaces imputationClass='stable-absence' 6. static-record absent entirely → coverage=0, NOT impute 7. Cohort determinism — identical inputs → identical scores 8. Different water-profile inputs → different scores (rules out regional-default hypothesis) Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail - `npm run test:data` — 6711 pass / 0 fail (PR 5.2's 9 + PR 5.3's 8 = 17 new stacked) - `npm run typecheck` / `typecheck:api` — green - `npm run lint` / `lint:md` — clean
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…x comment math Addresses 3 P2 Greptile findings on #3374 — all variations of the same root cause: the test fixture + doc described two different code paths that coincidentally both produce ~53 for GCC inputs. Changes 1. GCC anchor test now drives the IMPUTE branch (`fao: null`), matching what the static seeder emits for GCC in production. The else branch (`fao: { peopleInCrisis: 0 }`) happens to converge on ~52.94 by coincidence but is NOT the live code path for GCC. 2. Doc finding #4 updated to show the IMPUTE-branch math `(88×0.6 + 0×0.4) / 1.0 = 52.8 → 53` and explicitly notes the else-branch convergence as a coincidence — not the construct's intent. 3. Comment math off-by-one fix at line 107: (88×0.6 + 80×0.4) / (0.6+0.4) = 52.8 + 32.0 = 84.8 → 85 (was incorrectly stated as 85.6 → 86) Test assertion `>= 80 && <= 90` still accepts 85 so behaviour is unchanged; this was a comment-only error that would have misled anyone reproducing the math by hand. Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail (IMPUTE-branch anchor test produces 53 as expected) - `npm run lint:md` — clean Also rebased onto updated #3373 (which landed a backtick-escape fix).
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…istic GCC identity) PR 5.3 of cohort-audit plan 2026-04-24-002. Stacked on PR 5.2 (#3373) so the known-limitations.md section append is additive. Read-only static audit of scoreFoodWater. Findings 1. The observed GCC-all-score-53 is CONSTRUCT-DETERMINISTIC, not a regional-default leak. Pinned mathematically: - IPC/HDX doesn't publish active food-crisis data for food-secure states → scorer's fao-null branch imputes IMPUTE.ipcFood=88 (class='stable-absence', cov=0.7) at combined weight 0.6 - WB indicator ER.H2O.FWST.ZS (labelled 'water stress') for GCC is EXTREME (KW ~3200%, BH ~3400%, UAE ~2080%, QA ~770%) — all clamp to sub-score 0 under the scorer's lower-better 0..100 normaliser at weight 0.4 - Blended with peopleInCrisis=0 (fao block present with zero): (100 * 0.45 + 0 * 0.4) / (0.45 + 0.4) = 45 / 0.85 ≈ 53 Every GCC country has the same inputs → same outputs. That's construct math, not a regional lookup. 2. Indicator-keyword routing is code-correct. `'water stress'`, `'withdrawal'`, `'dependency'` route to lower-better; `'availability'`, `'renewable'`, `'access'` route to higher-better; unrecognized indicators fall through to a value-range heuristic with a WARN log. 3. No bug or methodology decision required. The 53-all-GCC output is a correct summary statement: "non-crisis food security + severe water-withdrawal stress." A future construct decision might split foodWater into separate food and water dims so one saturated sub-signal doesn't dominate the combined dim for desert economies — but that's a construct redesign, not a bug. Shipped - `docs/methodology/known-limitations.md` — extended with a new section documenting the foodWater audit findings, the exact blend math that yields ~53 for GCC, cohort-determinism vs regional-default, and a follow-up data-side spot-check list gated on API-key access. - `tests/resilience-foodwater-field-mapping.test.mts` — 8 new regression-guard tests: 1. indicator='water stress' routes to lower-better 2. GCC extreme-withdrawal anchor (value=2000 → blended score 53) 3. indicator='renewable water availability' routes to higher-better 4. fao=null with static record → imputes 88; imputationClass=null because observed AQUASTAT wins (weightedBlend T1.7 rule) 5. fully-imputed (fao=null + aquastat=null) surfaces imputationClass='stable-absence' 6. static-record absent entirely → coverage=0, NOT impute 7. Cohort determinism — identical inputs → identical scores 8. Different water-profile inputs → different scores (rules out regional-default hypothesis) Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail - `npm run test:data` — 6711 pass / 0 fail (PR 5.2's 9 + PR 5.3's 8 = 17 new stacked) - `npm run typecheck` / `typecheck:api` — green - `npm run lint` / `lint:md` — clean
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…x comment math Addresses 3 P2 Greptile findings on #3374 — all variations of the same root cause: the test fixture + doc described two different code paths that coincidentally both produce ~53 for GCC inputs. Changes 1. GCC anchor test now drives the IMPUTE branch (`fao: null`), matching what the static seeder emits for GCC in production. The else branch (`fao: { peopleInCrisis: 0 }`) happens to converge on ~52.94 by coincidence but is NOT the live code path for GCC. 2. Doc finding #4 updated to show the IMPUTE-branch math `(88×0.6 + 0×0.4) / 1.0 = 52.8 → 53` and explicitly notes the else-branch convergence as a coincidence — not the construct's intent. 3. Comment math off-by-one fix at line 107: (88×0.6 + 80×0.4) / (0.6+0.4) = 52.8 + 32.0 = 84.8 → 85 (was incorrectly stated as 85.6 → 86) Test assertion `>= 80 && <= 90` still accepts 85 so behaviour is unchanged; this was a comment-only error that would have misled anyone reproducing the math by hand. Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail (IMPUTE-branch anchor test produces 53 as expected) - `npm run lint:md` — clean Also rebased onto updated #3373 (which landed a backtick-escape fix).
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…istic GCC identity) (#3374) * docs(resilience): PR 5.3 — foodWater scorer audit (construct-deterministic GCC identity) PR 5.3 of cohort-audit plan 2026-04-24-002. Stacked on PR 5.2 (#3373) so the known-limitations.md section append is additive. Read-only static audit of scoreFoodWater. Findings 1. The observed GCC-all-score-53 is CONSTRUCT-DETERMINISTIC, not a regional-default leak. Pinned mathematically: - IPC/HDX doesn't publish active food-crisis data for food-secure states → scorer's fao-null branch imputes IMPUTE.ipcFood=88 (class='stable-absence', cov=0.7) at combined weight 0.6 - WB indicator ER.H2O.FWST.ZS (labelled 'water stress') for GCC is EXTREME (KW ~3200%, BH ~3400%, UAE ~2080%, QA ~770%) — all clamp to sub-score 0 under the scorer's lower-better 0..100 normaliser at weight 0.4 - Blended with peopleInCrisis=0 (fao block present with zero): (100 * 0.45 + 0 * 0.4) / (0.45 + 0.4) = 45 / 0.85 ≈ 53 Every GCC country has the same inputs → same outputs. That's construct math, not a regional lookup. 2. Indicator-keyword routing is code-correct. `'water stress'`, `'withdrawal'`, `'dependency'` route to lower-better; `'availability'`, `'renewable'`, `'access'` route to higher-better; unrecognized indicators fall through to a value-range heuristic with a WARN log. 3. No bug or methodology decision required. The 53-all-GCC output is a correct summary statement: "non-crisis food security + severe water-withdrawal stress." A future construct decision might split foodWater into separate food and water dims so one saturated sub-signal doesn't dominate the combined dim for desert economies — but that's a construct redesign, not a bug. Shipped - `docs/methodology/known-limitations.md` — extended with a new section documenting the foodWater audit findings, the exact blend math that yields ~53 for GCC, cohort-determinism vs regional-default, and a follow-up data-side spot-check list gated on API-key access. - `tests/resilience-foodwater-field-mapping.test.mts` — 8 new regression-guard tests: 1. indicator='water stress' routes to lower-better 2. GCC extreme-withdrawal anchor (value=2000 → blended score 53) 3. indicator='renewable water availability' routes to higher-better 4. fao=null with static record → imputes 88; imputationClass=null because observed AQUASTAT wins (weightedBlend T1.7 rule) 5. fully-imputed (fao=null + aquastat=null) surfaces imputationClass='stable-absence' 6. static-record absent entirely → coverage=0, NOT impute 7. Cohort determinism — identical inputs → identical scores 8. Different water-profile inputs → different scores (rules out regional-default hypothesis) Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail - `npm run test:data` — 6711 pass / 0 fail (PR 5.2's 9 + PR 5.3's 8 = 17 new stacked) - `npm run typecheck` / `typecheck:api` — green - `npm run lint` / `lint:md` — clean * fix(resilience): PR 5.3 review — pin IMPUTE branch for GCC anchor; fix comment math Addresses 3 P2 Greptile findings on #3374 — all variations of the same root cause: the test fixture + doc described two different code paths that coincidentally both produce ~53 for GCC inputs. Changes 1. GCC anchor test now drives the IMPUTE branch (`fao: null`), matching what the static seeder emits for GCC in production. The else branch (`fao: { peopleInCrisis: 0 }`) happens to converge on ~52.94 by coincidence but is NOT the live code path for GCC. 2. Doc finding #4 updated to show the IMPUTE-branch math `(88×0.6 + 0×0.4) / 1.0 = 52.8 → 53` and explicitly notes the else-branch convergence as a coincidence — not the construct's intent. 3. Comment math off-by-one fix at line 107: (88×0.6 + 80×0.4) / (0.6+0.4) = 52.8 + 32.0 = 84.8 → 85 (was incorrectly stated as 85.6 → 86) Test assertion `>= 80 && <= 90` still accepts 85 so behaviour is unchanged; this was a comment-only error that would have misled anyone reproducing the math by hand. Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail (IMPUTE-branch anchor test produces 53 as expected) - `npm run lint:md` — clean Also rebased onto updated #3373 (which landed a backtick-escape fix).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 5.2 of the cohort-audit plan (
docs/plans/2026-04-24-002-fix-resilience-cohort-ranking-structural-audit-plan.md). Read-only static audit of the UNHCR displacement field mapping consumed byscoreSocialCohesion,scoreBorderSecurity, andscoreStateContinuity.Parallel / non-blocking to PR 1 (#3372) and PR 3A (not yet started). Ships one new methodology doc + 9 regression-guard tests. No scorer behaviour changes.
Findings
1. Field mapping is code-correct
The plan's concern — "does
totalDisplacedinadvertently include labor migrants?" — is negative at the source. The UNHCR Population API doesn't publish labor migrant data at all; it covers only four categories (refugees, asylum seekers, IDPs, stateless), all of whichscripts/seed-displacement-summary.mjssums correctly intototalDisplacedandhostTotal.Labor-migrant-dominated cohorts (GCC states, Singapore, Malaysia) legitimately register as "no UNHCR footprint" — that's UNHCR semantics, not a bug. If we want demographic-pressure-from-foreign-born-populations as a resilience indicator, it needs a SEPARATE data source (UN DESA International Migrant Stock) + a separate dimension.
2. NEW finding —
scoreBorderSecurity??fallback is effectively dead codeAt
_dimension-scorers.ts:1412:Two compounding reasons it never fires:
safeNum(null)returns0, notnull. JavaScript'sNumber(null) === 0(whileNumber(undefined) === NaN). SosafeNumclassifiesnullas finite-zero, and??only falls back on null/undefined.scripts/seed-displacement-summary.mjsALWAYS writeshostTotal: 0explicitly for origin-only countries (lines 141-144). There is no production shape wherehostTotalis undefined.Observable consequence: origin-only high-outflow countries — Syria (~7M displaced), Venezuela (~6M), Afghanistan (~5M), Ukraine during peak — score
100onborderSecurity's displacement sub-component (35% of the dim blend). The outflow signal is effectively silenced.3. Not fixing in this PR
A one-line change (
||orhostTotal > 0 ? hostTotal : totalDisplaced) would flip borderSecurity for ~6 high-outflow origin countries by a material amount — a methodology change, not a pure bug-fix. Belongs in a construct-decision PR with before/after cohort snapshots, not bundled into an audit doc. Documented as a follow-up inknown-limitations.md.What shipped
docs/methodology/known-limitations.md(new) — "Displacement field-mapping" section covering source semantics, the GCC labor-migrant implication, the??short-circuit finding, and the follow-up audit list (11 countries for a live-data spot-check against UNHCR Refugee Data Finder — gated on API-key access).tests/resilience-displacement-field-mapping.test.mts(new) — 9 regression-guard tests pinning:scoreBorderSecuritybehaviour (Syria-pattern scores 100)??fallback case (undefined hostTotal DOES fallback; never emitted in prod)safeNum(null) === 0numeric-coercion contractstable-absenceimputation class for UNHCR-absent countriesscoreStateContinuityreadstotalDisplacedorigin-sideIntent: any future construct decision that flips
scoreBorderSecuritysemantics must update both the test ANDknown-limitations.mdin the same commit.Testing
npx tsx --test tests/resilience-displacement-field-mapping.test.mts— 9 pass / 0 failnpm run test:data— 6703 pass / 0 failnpm run typecheck/typecheck:api— greennpm run lint/lint:md— no warnings on new filesPost-Deploy Monitoring & Validation
No additional operational monitoring required: pure docs + tests. Zero behaviour change for the live scorer, no new endpoints, crons, flags, or Redis writes.Follow-ups (not in this PR)
scoreBorderSecurityreflect origin-outflow pressure, host-inflow pressure only, or both? Needs a separate PR with cohort snapshots.known-limitations.md). RequiresWORLDMONITOR_API_KEY/ Redis read access.Related
docs/plans/2026-04-24-002-fix-resilience-cohort-ranking-structural-audit-plan.md§PR 5.2server/worldmonitor/resilience/v1/_dimension-scorers.tslines 843, 1383, 1412, 1765🤖 Generated with Claude Opus 4.7 (1M context) via Claude Code + Compound Engineering v2.49.0
Co-Authored-By: Claude Opus 4.7 (1M context) [email protected]