feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read#3385
Merged
Conversation
Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003),
usable by any cohort seeder whose consumer needs bundle-level freshness.
Two coupled changes:
1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every
spawned child. All siblings in a single bundle run share one value
(captured at `runBundle` start, not spawn time). Consumers use this
to detect stale peer keys — if a peer's seed-meta predates the
current bundle run, fall back to a hard default rather than read
a cohort-peer's last-week output.
2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')`
handler that releases the acquired lock and extends existing-data
TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on
section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without
this handler the `finally` path never runs on SIGKILL, leaving
the 30-min acquireLock reservation in place until its own TTL
expires — the next cron tick silently skips the resource.
Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR
#3128 root cause).
Tests added:
- bundle-runner env injection (value within run bounds)
- sibling sections share the same timestamp (critical for the
consumer freshness guard)
- runSeed SIGTERM path: exit 143 + cleanup log
- process.once contract: second SIGTERM does not re-enter handler
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Greptile SummaryThis PR replaces the hand-curated YAML re-export-share manifest with a live Comtrade API seeder ( Confidence Score: 4/5Safe to merge; both findings are P2 and the fallback paths are robust. Only P2 findings: a potential row double-counting in parseComtradeFlowResponse (ratio likely correct in practice given empirical validation) and a wasted sleep on the last 429/5xx retry. The freshness guard, credential-leak check, and test coverage are well-designed. scripts/seed-recovery-reexport-share.mjs — parseComtradeFlowResponse row-summing semantics and retry backoff on final attempt. Important Files Changed
Sequence DiagramsequenceDiagram
participant BR as _bundle-runner.mjs
participant RS as seed-recovery-reexport-share.mjs
participant CT as UN Comtrade API
participant Redis
participant SW as seed-sovereign-wealth.mjs
BR->>BR: set BUNDLE_RUN_STARTED_AT_MS
BR->>RS: spawn (timeout 300s)
loop per cohort member (AE, PA)
RS->>CT: GET flowCode=M (header auth)
CT-->>RS: rows or 429 backoff retry
RS->>CT: GET flowCode=RX (header auth)
CT-->>RS: rows or 429 backoff retry
RS->>RS: parseComtradeFlowResponse / computeShareFromFlows / clampShare
end
RS->>RS: credential-leak guard check
RS->>Redis: SET reexport-share:v1 (TTL 35d)
RS->>Redis: SET seed-meta reexport-share (fetchedAt ms)
BR->>SW: spawn (timeout 600s)
SW->>Redis: GET reexport-share:v1
SW->>Redis: GET seed-meta reexport-share
SW->>SW: bundle-freshness guard
alt fetchedAt >= bundleStartMs
SW->>SW: apply share adjustment netImports = gross x (1 - share)
else stale or missing
SW->>SW: fallback to gross imports
end
SW->>Redis: SET sovereign-wealth:v1
Reviews (1): Last reviewed commit: "feat(resilience): Comtrade-backed re-exp..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+248
to
+258
| continue; | ||
| } | ||
|
|
||
| countries[iso2] = { | ||
| reexportShareOfImports: clamped, | ||
| year: picked.year, | ||
| reexportsUsd: picked.reexportsUsd, | ||
| grossImportsUsd: picked.importsUsd, | ||
| source: 'comtrade', | ||
| sources: [ | ||
| auditSafeSourceUrl(reporterCode, 'RX', years), |
Contributor
There was a problem hiding this comment.
parseComtradeFlowResponse sums every row unconditionally. According to the HHI seeder's comment (which filters partnerCode=0 to avoid double-counting), Comtrade can return both a world-aggregate row (partnerCode='0') and individual partner rows for the same year. If that happens here, primaryValue for each partner would be counted once directly and again via the world-total row, inflating both rxByYear and mByYear by a similar factor.
Because the same inflation affects both numerator and denominator the resulting share = RX/M is likely correct in practice (confirmed by the AE 35.5% empirical result). However, the test at line 1144 explicitly asserts that both a partnerCode='0' world row and a partnerCode='842' US row contribute to the same year's sum — which would be incorrect if Comtrade's TOTAL aggregate queries return both row types simultaneously.
Consider filtering to only partnerCode=0 rows (or only non-zero rows) once the actual API response shape is confirmed, or add a comment documenting why summing all rows is safe for cmdCode=TOTAL queries.
Comment on lines
+199
to
+210
| return { year, share: rawShare, reexportsUsd, importsUsd }; | ||
| } | ||
|
|
||
| /** | ||
| * Clamp a raw share into the material-and-safe range. Returns null for | ||
| * sub-floor shares (caller omits the country); caps at MAX_SHARE_CAP | ||
| * for above-ceiling shares. Pure function — exported for tests. | ||
| * | ||
| * @param {number} rawShare | ||
| * @returns {number | null} clamped share, or null if sub-floor | ||
| */ | ||
| export function clampShare(rawShare) { |
Contributor
There was a problem hiding this comment.
For 429 and 5xx responses, the code unconditionally sleeps backoffMs and then calls continue. On the last attempt (attempt === RETRY_MAX_ATTEMPTS = 3), the loop's update expression increments attempt to 4, the condition fails, and the function falls through to the post-loop return. The sleep at the final 429 attempt (6 s) and 5xx attempt (15 s) is wasted time — there is no subsequent request.
if (resp.status === 429) {
const backoffMs = 2000 * attempt;
if (attempt < RETRY_MAX_ATTEMPTS) {
console.warn(`...`);
await sleep(backoffMs);
}
continue;
}
if (resp.status >= 500) {
const backoffMs = 5000 * attempt;
if (attempt < RETRY_MAX_ATTEMPTS) {
console.warn(`...`);
await sleep(backoffMs);
}
continue;
}With a 2-country cohort this can add up to 42 wasted seconds (2 countries × 2 flow codes × 10.5 s average) against the 300 s bundle timeout.
Addresses PR #3384 review findings (todos 256, 257, 259, 260): #256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed at runSeed entry and armed through every `process.exit` path; could race `emptyDataIsFailure: true` strict-floor exits (IMF-External, WB-bulk) and extend seed-meta TTL when the contract forbids it — silently re-masking 30-day outages. Now the handler is attached immediately before `withRetry(fetchFn)` and removed in a try/finally that covers all fetch-phase exit branches. #257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper. Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc describing the bundle-freshness contract. Fleet-wide helper so the next consumer seeder imports instead of rediscovering the idiom. #259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key ops (`releaseLock` + `extendExistingTtl`). Serialising compounded Upstash latency during the exact failure mode (Redis degraded) this handler exists to handle, risking breach of the 5s SIGKILL grace. #260 (P2) — `_bundle-runner.mjs` asserts topological order on optional `dependsOn` section field. Throws on unknown-label refs and on deps appearing at a later index. Fleet-wide contract replacing the previous prose-comment ordering guarantee. Tests added/updated: - New: SIGTERM handler removed after fetchFn completes (narrowed-scope contract — post-fetch SIGTERM must NOT trigger TTL extension) - New: dependsOn unknown-label + out-of-order + happy-path (3 tests) Full test suite: 6,866 tests pass (+4 net).
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…, pin freshness boundary Addresses PR #3385 review findings: #257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared `getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the prereq commit) instead of its own `getBundleStartMs`. Single source of truth for the bundle-freshness contract. #258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the canonical `pathToFileURL(process.argv[1]).href === import.meta.url` form instead of basename-suffix matching. Handles symlinks, case- different paths on macOS HFS+, and Windows path separators without string munging. #260 (P2) consumer — Sovereign-Wealth declares `dependsOn: ['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq commit) now enforces topological order on load and throws on violation — replaces the previous prose-comment ordering contract. #261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis- reexport-share.test.mts` pinning the inclusive-boundary semantic: `fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards against a future refactor to `<=` that would silently reject peers writing at the very first millisecond of the bundle run. Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net).
koala73
force-pushed
the
feat/reexport-share-comtrade-seeder
branch
from
April 24, 2026 19:50
42295c4 to
c25067c
Compare
Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback regressed standalone (non-bundle) runs. A consumer seeder invoked manually just after its peer wrote `fetchedAt = (now - 5s)` would see `bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope as "stale", and fall back to defaults — defeating the point of the peer-read path outside the bundle. Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps the freshness gate scoped to its real purpose (across-bundle-tick staleness) and lets standalone runs skip the gate entirely. Consumers check `bundleStartMs != null` before applying the comparison; see the companion `seed-sovereign-wealth.mjs` change on the stacked PR.
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…, pin freshness boundary Addresses PR #3385 review findings: #257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared `getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the prereq commit) instead of its own `getBundleStartMs`. Single source of truth for the bundle-freshness contract. #258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the canonical `pathToFileURL(process.argv[1]).href === import.meta.url` form instead of basename-suffix matching. Handles symlinks, case- different paths on macOS HFS+, and Windows path separators without string munging. #260 (P2) consumer — Sovereign-Wealth declares `dependsOn: ['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq commit) now enforces topological order on load and throws on violation — replaces the previous prose-comment ordering contract. #261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis- reexport-share.test.mts` pinning the inclusive-boundary semantic: `fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards against a future refactor to `<=` that would silently reject peers writing at the very first millisecond of the bundle run. Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net).
koala73
force-pushed
the
feat/reexport-share-comtrade-seeder
branch
from
April 24, 2026 19:59
c25067c to
2702466
Compare
Greptile review P2 on PR #3384: the existing test only asserted exit code + log line, not that the Redis ops were actually issued. The log claim was ahead of the test. Fixture now logs every Upstash fetch call's shape (EVAL / pipeline- EXPIRE / other) to stderr. Test asserts: - >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua script on the lock key) - >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical + seed-meta keys) - The EVAL body carries the runSeed-generated runId (proves it's THIS run's release, not a phantom op) - The EXPIRE pipeline touches both the canonicalKey AND the seed-meta key (proves the keys[] array was built correctly including the extraKeys merge path) Full test suite: 6,866 tests pass, typecheck clean.
…read Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md Motivating case. Before this PR, the SWF `rawMonths` denominator for the `sovereignFiscalBuffer` dimension used GROSS annual imports for every country. For re-export hubs (goods transiting without domestic settlement), this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports = gross × (1 − reexport_share). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and never populated, so the correction never applied and the cohort audit showed no movement. Gap #2 (this PR). Two coupled changes to make the correction actually apply: 1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`). Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and `flowCode=M` (imports) per cohort member, compute share = RX/M at the latest co-populated year, clamp to [0.05, 0.95], publish the envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription key never reaches URL/logs/Redis. `maxRecords=250000` cap with truncation detection. Sequential + retry-on-429 with backoff. Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0): ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. 2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`). Swaps `loadReexportShareByCountry()` (YAML) for `loadReexportShareFromRedis()` (Redis key written by #1). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder's `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback to gross imports rather than apply last-month's stale share. Health registries. Both new keys registered in BOTH `api/health.js` SEED_META (60-day alert threshold) and `api/seed-health.js` SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match. Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min worst-case). Ordering preserved: Reexport-Share before Sovereign- Wealth so the SWF seeder reads a freshly-written key in the same cron tick. Deletions. YAML + loader + 7 obsolete loader tests removed; single source of truth is now Comtrade → Redis. Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm) which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed SIGTERM cleanup. This PR's bundle-freshness guard depends on that env variable. Tests (19 new, 7 deleted, +12 net): - Pure math: parseComtradeFlowResponse, computeShareFromFlows, clampShare, declareRecords + credential-leak source scan (15) - Integration (Gap #2 regression guards): SWF seeder loadReexport ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5) - Health registry dual-registry drift guard — scoped to this PR's keys, respecting pre-existing asymmetry (4) - Bundle-ordering + timeout assertions (2) Phase 0 cohort validation committed to plan. Full test suite passes: 6,881 tests.
…, pin freshness boundary Addresses PR #3385 review findings: #257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared `getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the prereq commit) instead of its own `getBundleStartMs`. Single source of truth for the bundle-freshness contract. #258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the canonical `pathToFileURL(process.argv[1]).href === import.meta.url` form instead of basename-suffix matching. Handles symlinks, case- different paths on macOS HFS+, and Windows path separators without string munging. #260 (P2) consumer — Sovereign-Wealth declares `dependsOn: ['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq commit) now enforces topological order on load and throws on violation — replaces the previous prose-comment ordering contract. #261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis- reexport-share.test.mts` pinning the inclusive-boundary semantic: `fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards against a future refactor to `<=` that would silently reject peers writing at the very first millisecond of the bundle run. Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net).
…l required Review catch: the previous `bundleStartMs = Date.now()` fallback made standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any previously-seeded re-export-share meta as "stale" — even when the operator ran the Reexport seeder milliseconds beforehand. Defeated the point of the peer-read path outside the bundle. With `getBundleRunStartedAtMs()` now returning null outside a bundle (companion commit on the prereq branch), the consumer only applies the freshness gate when `bundleStartMs != null`. Standalone runs accept any `fetchedAt` — the operator is responsible for ordering. Two guards survive the change: - Meta MUST exist (absence = peer-outage fail-safe, both modes) - In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS` Two new tests pin both modes: - standalone: accepts meta written 10 min before this process started - standalone: still rejects missing meta (peer-outage fail-safe survives gate bypass) Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net).
…try sleep Greptile review of PR #3385 flagged two P2s in the Comtrade seeder. Finding #3 (parseComtradeFlowResponse double-count risk): `cmdCode=TOTAL` without a partner filter currently returns only world-aggregate rows in practice — but `parseComtradeFlowResponse` summed every row unconditionally. A future refactor adding per- partner querying would silently double-count (world-aggregate row + partner-level rows for the same year), cutting the derived share in half with no test signal. Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches current empirical behavior (aggregate-only responses) and makes the construct robust to a future partner-level query. Finding #4 (wasted backoff on final retry): 429 and 5xx branches slept `backoffMs` before `continue`, but on `attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately after — the sleep was pure waste. Added early-return (parallel to the existing pattern in the network-error catch branch) so the final attempt exits the retry loop at the first non-success response without extra latency. Tests: - 3 new `parseComtradeFlowResponse` variants: world-only filter, numeric-0 partnerCode shape, rows without partnerCode field - Existing tests updated: the double-count assertion replaced with a "per-partner rows must NOT sum into the world-aggregate total" assertion that pins the new contract Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).
koala73
force-pushed
the
feat/reexport-share-comtrade-seeder
branch
from
April 24, 2026 20:10
2702466 to
5662c24
Compare
Owner
Author
|
Addressed both P2s from the Greptile review plus a follow-up regression that surfaced during my own review pass: Double-counting when world-aggregate and partner rows co-exist (commit Unnecessary backoff sleep on final retry (commit Standalone-regression in freshness gate (commits Full suite: 6,890 tests pass, typecheck clean. |
This was referenced Apr 25, 2026
koala73
added a commit
that referenced
this pull request
Apr 28, 2026
…+ net-imports parity (v17.1) (#3482) * feat(resilience): plan 002 §U8 — methodology rewrite, doc-parity test, widget polish + net-imports parity for liquidReserveAdequacy (v17.1) Closes the plan 2026-04-26-002 sequence. Five things land together: 1. **§U8.1 — net-imports denominator parity for liquidReserveAdequacy.** PR #3380 shipped `computeNetImports(grossImports, reexportShareOfImports)` for `sovereignFiscalBuffer` via the SWF seeder, sourced from `resilience:recovery:reexport-share:v1` (Comtrade-backed via PR #3385). The same correction was structurally needed on the sibling `liquidReserveAdequacy` dimension — re-export hubs (AE 35.5%, PA similar) consume WB `FI.RES.TOTL.MO` which is computed at WB source against gross imports, double-counting goods that flow through without settling as domestic consumption. v17.1 multiplies WB's pre-computed months by `1/(1−reexportShare)` for hub countries (algebraic inverse of dividing the denominator) — yields the same adjusted-months a custom `reserves/(net-imports/12)` calc would, without re-fetching raw `FI.RES.TOTL.CD` + `BM.GSR.GNFS.CD` series. Non-hub countries unchanged (status quo). Expected impact at next ranking refresh: AE liquidReserveAdequacy 38 → ~64 (a +26-point dim swing); PA similar magnitude. 2. **§U8 — methodology doc comprehensive v17 changelog.** Single coherent narrative covering the universe + coverage rebuild: source-comprehensiveness flag, coverage penalty multiplier, per-capita normalization with 0.5M tiny-state floor, headline- eligible gate, symmetric cache-hit gate filtering. Anchored to the live `resilience:ranking:v17` cohort numbers captured 2026-04-28 post-#3477 merge: median(Nordics)=78.52 vs median(GCC)=70.53 (+7.98pt gap), min(G7)=64.31 vs max(LIC)=53.73 (+10.58pt gap), 1 microstate (MO at #4) in top-20. Documents 3 known open construct gaps honestly: economic-complexity / industrial-base indicator, importConcentration coverage gap on AE, cyberDigital transient zeros. 3. **§U8 — new doc-parity test (`tests/resilience-doc-parity.test.mts`).** Asserts the doc's prose claims match `_shared.ts` (cache prefix constants), `RESILIENCE_DOMAIN_ORDER` (6 domains), and `RESILIENCE_DIMENSION_ORDER` − `RESILIENCE_RETIRED_DIMENSIONS` (20 active dimensions). Caught real drift: Redis keys table named v11 prefixes despite live v17, and the headline "19 dimensions" claim was off-by-one after PR 2 §3.4 added liquidReserveAdequacy + sovereignFiscalBuffer. Fixed both as part of this commit. 4. **§U8 — widget polish.** `formatResilienceConfidence` now renders "Outside headline ranking" when `headlineEligible: false`, distinct from "Low confidence — sparse data" when `lowConfidence: true`. The two reasons are different and analysts should see them as different. Order matters: lowConfidence is more specific so it wins when both flags fire. New test pins the precedence. 5. **MDX-unsafe `<digit/letter` sweep + `lint:md`.** Both clean. 89 dimension-scorer tests + 36 widget tests + 5 doc-parity tests + 41 ranking + headline-gate tests all green (171 total, 0 fail). * fix(resilience): demote v17.1 changelog H4 to bold-paragraph (methodology-lint compat) CI methodology-lint test (T1.8) requires every H4 in docs/methodology/country-resilience-index.mdx to map to a known scorer dimension via HEADING_TO_DIMENSION — that's how the doc-vs- registry parity gate works. The v17.1 changelog sub-section I added (`#### v17.1 — Net-imports denominator parity for liquidReserveAdequacy (U8.1)`) is a non-dimension subsection and tripped the linter. Demoted to a bold lead-in paragraph (`**v17.1 — …**`). Same content, same prose, just no H4. The lint comment explicitly anticipates this case ("if a future edit adds a legitimate non-dimension H4, either upgrade it to H3 or add an explicit allowlist") — bold-paragraph is the cleanest of the three options and matches how Mintlify renders sub-changelog notes elsewhere in the doc. Local re-run: methodology-lint + doc-parity 10/10 pass; full test:data suite 7606/7606 pass, 0 fail. * fix(resilience): plan 002 §U8 review fixes (P1 cache bump v17→v18, P2 doc TTL/dim drift, P2 stale-count tightening) Three review findings + a Greptile P2 perf note. All addressed locally. P1 — cache invalidation missing for the §U8.1 scorer-formula change. liquidReserveAdequacy now adjusts reserveMonths by re-export share for AE/PA, but the prior commit kept score/ranking at v17 and history at v12. The `_formula` tag is binary 'd6'|'pc' and does NOT detect intra- 'd6' scorer changes — without this bump, cached v17 AE/PA scores (gross-imports-denominated) would continue to serve until TTL expiry, defeating the construct fix this PR delivers. Bumped: - RESILIENCE_SCORE_CACHE_PREFIX v17 → v18 (_shared.ts:164) - RESILIENCE_RANKING_CACHE_KEY v17 → v18 (_shared.ts:235) - RESILIENCE_HISTORY_KEY_PREFIX v12 → v13 (_shared.ts:217) - script mirrors: seed-resilience-scores.mjs, backtest, validate, benchmark - api/health.js: resilienceRanking key History bumps in lockstep so the rolling 30-day window doesn't mix pre/post-fix points and manufacture a false "improving" trend on day one. Same pattern as PR 3A's v11→v12 lockstep for the SWF-side fix. P2 — Redis keys table TTL + write-gate prose drift. Doc claimed: resilience:ranking:v17 | 6 hours | "only when all countries are scored" Actual code: 12h TTL (RESILIENCE_RANKING_CACHE_TTL_SECONDS) and publishes when coverage >= RANKING_CACHE_MIN_COVERAGE = 0.75. Updated the table to reflect both. Same correction caught by the parity test that flagged the v11/v17 drift in the prior commit. P2 — stale "19 dimensions" claim still in the doc. The v2.0 scorecard Data row at line 750 still said "19 dimensions across 6 domains" because the parity test only required ONE correct mention. Tightened the test to fail on any "<N> dimensions" mention in the plausible-current-total band [15, 25] that doesn't equal activeCount or totalCount — historical sub-counts (e.g. v1.0's "13 dimensions", recovery pillar's "6 dimensions") are outside the band and stay untouched. Updated the stale line to "20 active dimensions across 6 domains (plus 2 structurally-retired dimensions)". Bonus — addressed a load of test-side cache-key literal duplications (plan 002 §U8 follow-up): - tests/resilience-ranking.test.mts: 27 hardcoded 'resilience:score:v17:XX' / 'resilience:history:v12:XX' literals swapped to ${RESILIENCE_SCORE_CACHE_PREFIX}/${RESILIENCE_HISTORY_KEY_PREFIX} template-literal references. - tests/resilience-handlers.test.mts: same. - tests/resilience-pillar-aggregation.test.mts: pinned-to-version assertion replaced with structural-shape regex. - tests/resilience-scores-seed.test.mjs: same. This was caught by PR #3477 review for the ranking key; the score and history prefixes had the same drift trap that resurfaced now. The proper fix per the project skill (proto-required-field-add-needs-cache- backfill Trap 5b) is to import the constants — never hardcode literals. Greptile P2 (per-country Redis read for reexport-share map) — verified benign: createMemoizedSeedReader caches by key, and _shared.ts:992's `sharedReader` is shared across all 196 countries in the warm path (getOrComputeResilienceScores), so reexport-share is fetched ONCE per ranking compute, not 196 times. Single-country handler creates a fresh memoized reader per request → 1 read per request. No fix needed. Verification: 214/214 resilience tests pass, 7606/7606 npm test:data pass, typecheck clean, lint:md clean.
fuleinist
pushed a commit
to fuleinist/worldmonitor
that referenced
this pull request
May 9, 2026
…+ net-imports parity (v17.1) (koala73#3482) * feat(resilience): plan 002 §U8 — methodology rewrite, doc-parity test, widget polish + net-imports parity for liquidReserveAdequacy (v17.1) Closes the plan 2026-04-26-002 sequence. Five things land together: 1. **§U8.1 — net-imports denominator parity for liquidReserveAdequacy.** PR koala73#3380 shipped `computeNetImports(grossImports, reexportShareOfImports)` for `sovereignFiscalBuffer` via the SWF seeder, sourced from `resilience:recovery:reexport-share:v1` (Comtrade-backed via PR koala73#3385). The same correction was structurally needed on the sibling `liquidReserveAdequacy` dimension — re-export hubs (AE 35.5%, PA similar) consume WB `FI.RES.TOTL.MO` which is computed at WB source against gross imports, double-counting goods that flow through without settling as domestic consumption. v17.1 multiplies WB's pre-computed months by `1/(1−reexportShare)` for hub countries (algebraic inverse of dividing the denominator) — yields the same adjusted-months a custom `reserves/(net-imports/12)` calc would, without re-fetching raw `FI.RES.TOTL.CD` + `BM.GSR.GNFS.CD` series. Non-hub countries unchanged (status quo). Expected impact at next ranking refresh: AE liquidReserveAdequacy 38 → ~64 (a +26-point dim swing); PA similar magnitude. 2. **§U8 — methodology doc comprehensive v17 changelog.** Single coherent narrative covering the universe + coverage rebuild: source-comprehensiveness flag, coverage penalty multiplier, per-capita normalization with 0.5M tiny-state floor, headline- eligible gate, symmetric cache-hit gate filtering. Anchored to the live `resilience:ranking:v17` cohort numbers captured 2026-04-28 post-koala73#3477 merge: median(Nordics)=78.52 vs median(GCC)=70.53 (+7.98pt gap), min(G7)=64.31 vs max(LIC)=53.73 (+10.58pt gap), 1 microstate (MO at #4) in top-20. Documents 3 known open construct gaps honestly: economic-complexity / industrial-base indicator, importConcentration coverage gap on AE, cyberDigital transient zeros. 3. **§U8 — new doc-parity test (`tests/resilience-doc-parity.test.mts`).** Asserts the doc's prose claims match `_shared.ts` (cache prefix constants), `RESILIENCE_DOMAIN_ORDER` (6 domains), and `RESILIENCE_DIMENSION_ORDER` − `RESILIENCE_RETIRED_DIMENSIONS` (20 active dimensions). Caught real drift: Redis keys table named v11 prefixes despite live v17, and the headline "19 dimensions" claim was off-by-one after PR 2 §3.4 added liquidReserveAdequacy + sovereignFiscalBuffer. Fixed both as part of this commit. 4. **§U8 — widget polish.** `formatResilienceConfidence` now renders "Outside headline ranking" when `headlineEligible: false`, distinct from "Low confidence — sparse data" when `lowConfidence: true`. The two reasons are different and analysts should see them as different. Order matters: lowConfidence is more specific so it wins when both flags fire. New test pins the precedence. 5. **MDX-unsafe `<digit/letter` sweep + `lint:md`.** Both clean. 89 dimension-scorer tests + 36 widget tests + 5 doc-parity tests + 41 ranking + headline-gate tests all green (171 total, 0 fail). * fix(resilience): demote v17.1 changelog H4 to bold-paragraph (methodology-lint compat) CI methodology-lint test (T1.8) requires every H4 in docs/methodology/country-resilience-index.mdx to map to a known scorer dimension via HEADING_TO_DIMENSION — that's how the doc-vs- registry parity gate works. The v17.1 changelog sub-section I added (`#### v17.1 — Net-imports denominator parity for liquidReserveAdequacy (U8.1)`) is a non-dimension subsection and tripped the linter. Demoted to a bold lead-in paragraph (`**v17.1 — …**`). Same content, same prose, just no H4. The lint comment explicitly anticipates this case ("if a future edit adds a legitimate non-dimension H4, either upgrade it to H3 or add an explicit allowlist") — bold-paragraph is the cleanest of the three options and matches how Mintlify renders sub-changelog notes elsewhere in the doc. Local re-run: methodology-lint + doc-parity 10/10 pass; full test:data suite 7606/7606 pass, 0 fail. * fix(resilience): plan 002 §U8 review fixes (P1 cache bump v17→v18, P2 doc TTL/dim drift, P2 stale-count tightening) Three review findings + a Greptile P2 perf note. All addressed locally. P1 — cache invalidation missing for the §U8.1 scorer-formula change. liquidReserveAdequacy now adjusts reserveMonths by re-export share for AE/PA, but the prior commit kept score/ranking at v17 and history at v12. The `_formula` tag is binary 'd6'|'pc' and does NOT detect intra- 'd6' scorer changes — without this bump, cached v17 AE/PA scores (gross-imports-denominated) would continue to serve until TTL expiry, defeating the construct fix this PR delivers. Bumped: - RESILIENCE_SCORE_CACHE_PREFIX v17 → v18 (_shared.ts:164) - RESILIENCE_RANKING_CACHE_KEY v17 → v18 (_shared.ts:235) - RESILIENCE_HISTORY_KEY_PREFIX v12 → v13 (_shared.ts:217) - script mirrors: seed-resilience-scores.mjs, backtest, validate, benchmark - api/health.js: resilienceRanking key History bumps in lockstep so the rolling 30-day window doesn't mix pre/post-fix points and manufacture a false "improving" trend on day one. Same pattern as PR 3A's v11→v12 lockstep for the SWF-side fix. P2 — Redis keys table TTL + write-gate prose drift. Doc claimed: resilience:ranking:v17 | 6 hours | "only when all countries are scored" Actual code: 12h TTL (RESILIENCE_RANKING_CACHE_TTL_SECONDS) and publishes when coverage >= RANKING_CACHE_MIN_COVERAGE = 0.75. Updated the table to reflect both. Same correction caught by the parity test that flagged the v11/v17 drift in the prior commit. P2 — stale "19 dimensions" claim still in the doc. The v2.0 scorecard Data row at line 750 still said "19 dimensions across 6 domains" because the parity test only required ONE correct mention. Tightened the test to fail on any "<N> dimensions" mention in the plausible-current-total band [15, 25] that doesn't equal activeCount or totalCount — historical sub-counts (e.g. v1.0's "13 dimensions", recovery pillar's "6 dimensions") are outside the band and stay untouched. Updated the stale line to "20 active dimensions across 6 domains (plus 2 structurally-retired dimensions)". Bonus — addressed a load of test-side cache-key literal duplications (plan 002 §U8 follow-up): - tests/resilience-ranking.test.mts: 27 hardcoded 'resilience:score:v17:XX' / 'resilience:history:v12:XX' literals swapped to ${RESILIENCE_SCORE_CACHE_PREFIX}/${RESILIENCE_HISTORY_KEY_PREFIX} template-literal references. - tests/resilience-handlers.test.mts: same. - tests/resilience-pillar-aggregation.test.mts: pinned-to-version assertion replaced with structural-shape regex. - tests/resilience-scores-seed.test.mjs: same. This was caught by PR koala73#3477 review for the ranking key; the score and history prefixes had the same drift trap that resurfaced now. The proper fix per the project skill (proto-required-field-add-needs-cache- backfill Trap 5b) is to import the constants — never hardcode literals. Greptile P2 (per-country Redis read for reexport-share map) — verified benign: createMemoizedSeedReader caches by key, and _shared.ts:992's `sharedReader` is shared across all 196 countries in the warm path (getOrComputeResilienceScores), so reexport-share is fetched ONCE per ranking compute, not 196 times. Single-country handler creates a fresh memoized reader per request → 1 read per request. No fix needed. Verification: 214/214 resilience tests pass, 7606/7606 npm test:data pass, typecheck clean, lint:md clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #3384 (the prereq
BUNDLE_RUN_STARTED_AT_MS/ SIGTERM-cleanup PR — this PR's bundle-freshness guard depends on that env var being populated by_bundle-runner.mjs).Motivating case. Before this PR, the SWF
rawMonthsdenominator for thesovereignFiscalBufferdimension used GROSS annual imports for every country. For re-export hubs, this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports =gross × (1 − reexportShareOfImports). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and the correction never applied — the cohort audit showed no movement.Two coupled changes here:
Comtrade-backed seeder (
scripts/seed-recovery-reexport-share.mjs, rewrite). FetchesflowCode=RX+flowCode=Mper cohort member, computesshare = RX/Mat the latest co-populated year, clamps to[0.05, 0.95], publishes Redis envelope (manifestVersion: 2). Header auth — subscription key never reaches URL/logs/Redis.maxRecords=250000cap with truncation detection. Sequential + retry-on-429 with backoff.Phase 0 empirical cohort validation (committed to plan doc):
['AE', 'PA']. Six candidates (SG, HK, NL, BE, MY, LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. AE at 35.5% share (2023) is the primary mover.SWF seeder reads from Redis (
scripts/seed-sovereign-wealth.mjs). SwapsloadReexportShareByCountry()(YAML, now deleted) forloadReexportShareFromRedis(). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder'sseed-metapredatesBUNDLE_RUN_STARTED_AT_MS, HARD fallback to gross imports rather than apply last-month's stale share.Health registries. Both new keys registered in BOTH
api/health.jsSEED_META (60-day alert) andapi/seed-health.jsSEED_DOMAINS (43200min interval).Bundle wiring. Reexport-Share timeout 60s → 300s (Comtrade + retry worst-case). Ordering preserved: Reexport-Share before Sovereign-Wealth so SWF reads a freshly-written key in the same cron tick.
Deletions. YAML + loader + 7 obsolete loader tests. Single source of truth is now Comtrade → Redis.
Test plan
parseComtradeFlowResponse,computeShareFromFlows,clampShare,declareRecords, credential-leak source scan)loadReexportShareFromRedis— fresh / absent / malformed / stale-meta / missing-meta). These are the Gap Add smart zoom visibility for dense map layers #2 regression guards.npm run test:datapasses (6,881 tests)npm run typecheckcleannpm run lint— no new errors (206 pre-existing warnings)Post-Deploy Monitoring & Validation
seed-bundle-resilience-recoveryservice — look for[reexport-share] AE: share=XX.X% at Y=YYYYper run; the seeder should successfully publish both AE (share ~35%) and PA (either published ~7% clamped, or omitted if Phase 0's 1.4% floor trip).GET resilience:recovery:reexport-share:v1 | jq '.countries'— expect{ AE: {...}, PA: {...} }or{ AE: {...} }depending on PA's current share.curl $API/api/health | jq '.seeds.recoveryReexportShare, .seeds.recoverySovereignWealth'— bothfetchedAt< 24h.node scripts/audit-resilience-cohorts.mjs— UAE sovereignFiscalBuffer score rises from ~72 → ~85.[seed-sovereign-wealth] re-export adjustment applied to 1 country/countries: AE share=0.35 gross=$941.1B net=$611.7B (source year 2023).reexport-share seed-meta NOT from this bundle runlog on every SWF run — indicates the Reexport-Share seeder is failing silently upstream. Check Railway logs for Comtrade errors. Safe state: SWF falls back to gross imports automatically (no scoring regression).🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via Claude Code + Compound Engineering v2.49.0