Skip to content

feat(wb-cohort): Sprint 4 follow-up — content-age for low-carbon + fossil-share#3603

Merged
koala73 merged 2 commits into
mainfrom
feat/health-readiness-content-age-sprint-4-cohort-fossil-low-carbon
May 8, 2026
Merged

feat(wb-cohort): Sprint 4 follow-up — content-age for low-carbon + fossil-share#3603
koala73 merged 2 commits into
mainfrom
feat/health-readiness-content-age-sprint-4-cohort-fossil-low-carbon

Conversation

@koala73

@koala73 koala73 commented May 5, 2026

Copy link
Copy Markdown
Owner

Sprint 4 cohort follow-up of the 2026-05-04 health-readiness probe plan. Migrates the two remaining WB resilience seeders that match power-reliability's shape. Branched off Sprint 1 (#3596) as a parallel sibling to #3597 / #3598 / #3599 / #3602.

Summary

  • New shared helper `scripts/_wb-country-dict-content-age-helpers.mjs`: `yearToEndOfYearMs`, `wbCountryDictContentMeta`. Three production seeders use the IDENTICAL per-country-dict shape — meets CLAUDE.md's "three similar lines" line for justifying abstraction.
  • Wires `seed-low-carbon-generation` (36mo budget) and `seed-fossil-electricity-share` (48mo budget) into the content-age probe.
  • Per-seeder budgets are NOT one-size-fits-all — verified against live WB API.

Per-seeder budgets — verified against live WB on 2026-05-05

Seeder Indicator(s) Live max year Fresh-arrival lag Budget Steady-state ceiling
seed-low-carbon-generation NUCL+RNEW+HYRO sum, MAX year of 3 2024 (NUCL/HYRO) ~17mo 36mo 30mo
seed-fossil-electricity-share EG.ELC.FOSL.ZS 2023 ~29mo 48mo 41mo

A naive cohort-wide budget would either false-positive on fossil-share (if 36mo) or be wastefully loose on low-carbon (if 48mo). Per-seeder constants are the correct response.

The "per-seeder budget separation" test pins this explicitly: a 41mo cache trips low-carbon but NOT fossil-share. Demonstrates that budgets aren't accidental — they reflect real upstream cadence.

Why share the helper this time (not in Sprint 2/3a/3b/4 power-reliability)

Three production seeders now use the IDENTICAL shape and IDENTICAL math. Per CLAUDE.md "three similar lines is better than a premature abstraction" — three is the line. `seed-power-reliability` keeps its own helper for now (PR #3602 is in review; backporting to the shared helper is a follow-up after merge to keep that PR's diff focused). The math is verifiably identical.

Renewables (RNEW.ZS) data-quality flag

Audit found EG.ELC.RNEW.ZS max year = 2021 in May 2026 (~53mo lag). Inside low-carbon-generation it's masked by MAX(NUCL, RNEW, HYRO), so content-age looks fine. But the underlying renewable share data is genuinely 5+ years stale. Not addressed in this PR — flagging as a separate data-quality concern for follow-up review (separate from the content-age probe contract).

Test plan

  • `node --test tests/wb-country-dict-content-age.test.mjs` → 15/15 pass
  • `node --test tests/wb-country-dict-content-age.test.mjs tests/seed-content-age-contract.test.mjs tests/health-content-age.test.mjs tests/seed-utils-empty-data-failure.test.mjs` → 47/47 pass
  • `npm run typecheck:api` clean
  • `npm run lint` clean
  • Fresh-arrival regression guards for both seeders (Sprint 3b lesson): pin natural fresh-arrival within budget
  • Steady-state regression guards for both seeders (Sprint 4 lesson): pin one-cycle-late case within budget
  • Catastrophic stall tests for both seeders: pin clearly-stale case past budget
  • Per-seeder budget separation test: 41mo cache trips low-carbon but not fossil-share
  • Neither seeder is in Dockerfile.relay (verified via grep) — no relay-COPY change needed
  • Post-merge: confirm `/api/health` snapshot shows `lowCarbonGeneration` and `fossilElectricityShare` with `contentAge` populated and `status: OK`
  • Post-merge: open a separate issue for the RNEW.ZS data-quality finding

Sprint 4 status after this PR

3 of 5 plan-listed annual indicators migrated. Plan's "Definition of done" §530 (≥1 annual seeder) was already satisfied by #3602; this PR rounds out the WB cohort.

@vercel

vercel Bot commented May 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment May 8, 2026 9:02am

Request Review

@greptile-apps

greptile-apps Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds a shared _wb-country-dict-content-age-helpers.mjs module and wires contentMeta/maxContentAgeMin into the seed-low-carbon-generation (36mo) and seed-fossil-electricity-share (48mo) seeders, completing the WB annual-indicator cohort for the content-age health probe.

  • New shared helper exports yearToEndOfYearMs and wbCountryDictContentMeta; clock-skew guard, null-entry handling, and per-country MAX/MIN derivation are correct and match the contract already used by seed-power-reliability.
  • Per-seeder budgets are intentionally different (36mo vs 48mo) to reflect each indicator's observed WB publication lag, and the budget-separation test explicitly pins the 41-month threshold that would false-positive on low-carbon but not fossil-share.
  • Test suite adds 15 regression-guard tests covering fresh-arrival, steady-state, catastrophic stall, and per-seeder separation cases.

Confidence Score: 4/5

The core content-age wiring is correct and safe to merge; only a misplaced import and a misleading test comment need attention.

The helper logic, budget math, and runSeed integration are all correct. The only issues found are an import declaration that appears after a const in seed-fossil-electricity-share.mjs (inconsistent with the sibling seeder's layout), and a test comment for the EDGE case that says 'past FIXED_NOW' when Dec 31 2026 is actually in the future relative to the test's fixed timestamp. Neither affects runtime behaviour.

scripts/seed-fossil-electricity-share.mjs (import ordering) and tests/wb-country-dict-content-age.test.mjs (EDGE comment).

Important Files Changed

Filename Overview
scripts/_wb-country-dict-content-age-helpers.mjs New shared helper exporting yearToEndOfYearMs and wbCountryDictContentMeta; logic is correct, skew-limit and null-guard handling are sound.
scripts/seed-fossil-electricity-share.mjs Wires contentMeta/maxContentAgeMin (48mo) into runSeed; correct budget for the indicator's observed lag, but import iso3ToIso2 is placed after the const declaration, inconsistent with sibling seeders.
scripts/seed-low-carbon-generation.mjs Wires contentMeta/maxContentAgeMin (36mo) into runSeed; import ordering is clean and budget rationale is well-documented.
tests/wb-country-dict-content-age.test.mjs Thorough test coverage across all helper edge cases and per-seeder budget regression guards; one inline comment for the EDGE (year=2026) case incorrectly describes Dec 31 2026 as 'past FIXED_NOW' when it is actually future-dated.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[seed-low-carbon-generation.mjs] -->|MAX year per country NUCL/RNEW/HYRO| B[wbCountryDictContentMeta]
    C[seed-fossil-electricity-share.mjs] -->|per-country latest year EG.ELC.FOSL.ZS| B
    B -->|newestItemAt / oldestItemAt| D[runSeed]
    D --> E{contentAgeOptedIn?}
    E -->|yes| F[writeFreshnessMetadata]
    F --> G[/api/health]
    G --> H{age > maxContentAgeMin?}
    H -->|yes| I[STALE_CONTENT]
    H -->|no| J[OK]
Loading

Reviews (1): Last reviewed commit: "feat(wb-cohort): Sprint 4 follow-up — co..." | Re-trigger Greptile

Comment thread tests/wb-country-dict-content-age.test.mjs
Comment thread scripts/seed-fossil-electricity-share.mjs Outdated
koala73 added a commit that referenced this pull request May 7, 2026
…rder)

P2 — `tests/wb-country-dict-content-age.test.mjs:79` — misleading inline
comment: read `// end-of-2026 = Dec 31 23:59:59 = past FIXED_NOW (May 5)`
but FIXED_NOW is 2026-05-05 and end-of-2026 is ~7 months in the FUTURE,
not past. The test logic is correct (the EDGE year IS excluded as
future-dated beyond skew tolerance) — only the comment was wrong.

P2 — `scripts/seed-fossil-electricity-share.mjs:30` — `import iso3ToIso2`
appeared on the line immediately after `const MAX_CONTENT_AGE_MIN`.
ES module `import`s are hoisted regardless of source order, but
interleaving with declarations confuses readers (code "looks" sequential
but the import actually executes first). Moved the import up alongside
the other top-of-module imports.

Both pure-text nits — no behavior change. typecheck clean; targeted
tests/wb-country-dict-content-age.test.mjs passes 15/15.
Base automatically changed from feat/health-readiness-content-age-sprint-1 to main May 8, 2026 08:43
koala73 added 2 commits May 8, 2026 12:55
…ssil-share

Sprint 4 cohort follow-up of the 2026-05-04 health-readiness probe plan.
Migrates the two remaining WB resilience seeders that match power-reliability's
shape: seed-low-carbon-generation.mjs and seed-fossil-electricity-share.mjs.
Branched off Sprint 1 (#3596) as a parallel sibling.

## Why a shared helper this time

Three production seeders now use the IDENTICAL per-country-dict shape
({countries: {ISO2: {value, year}}, seededAt}) with the IDENTICAL
contentMeta math (max-year selection + end-of-year UTC + 1h skew limit).
Per CLAUDE.md "three similar lines is better than a premature abstraction"
— three is exactly the line for justifying the abstraction now.

New `scripts/_wb-country-dict-content-age-helpers.mjs` exports:
  - yearToEndOfYearMs(year)
  - wbCountryDictContentMeta(data, nowMs?)

Each seeder imports it + brings its own MAX_CONTENT_AGE_MIN constant
inline (per-seeder budgets matter — see below). seed-power-reliability
keeps its own helper for now (PR #3602 is in review; backporting to the
shared helper is a follow-up after merge to keep that PR's diff focused).
The math is verifiably identical.

## Per-seeder budgets (NOT one-size-fits-all)

Verified against live WB API on 2026-05-05 — publication lags differ
across these "annual WB indicators":

  - low-carbon-generation (NUCL+RNEW+HYRO sum, MAX year of 3):
      max year = 2024 (driven by NUCL/HYRO; RNEW lags to 2021 but is
      masked by MAX-of-3 in the seeder's countries[iso2].year compute)
      → fresh-arrival lag ~17mo
      → 36mo budget (= 30mo steady-state ceiling + 6mo slack)
      → matches power-reliability exactly

  - fossil-electricity-share (EG.ELC.FOSL.ZS):
      max year = 2023 (NOT 2024 — slower-publishing indicator)
      → fresh-arrival lag ~29mo
      → 48mo budget (= 41mo steady-state ceiling + 7mo slack)

A naive cohort-wide budget would either false-positive on fossil-share
(if 36mo) or be wastefully loose on low-carbon (if 48mo). Per-seeder
constants are the correct response — each indicator's lag is empirically
different.

The "per-seeder budget separation" test pins this explicitly: a 41mo cache
trips low-carbon (36mo) but NOT fossil-share (48mo). Demonstrates that the
budgets aren't accidental — they reflect real upstream cadence differences.

## Renewables (RNEW.ZS) data-quality flag

Discovered during the audit: EG.ELC.RNEW.ZS max year = 2021 in May 2026,
~53mo lag. Inside low-carbon-generation it's masked by MAX(NUCL, RNEW,
HYRO), so content-age looks fine. But the underlying renewable share
data is genuinely 5+ years stale. Not addressed in this PR — flagging
as a separate data-quality concern for follow-up review.

## Verification

  - 15/15 wb-country-dict content-age tests pass (incl. fresh-arrival +
    steady-state regression guards for BOTH new seeders, plus a
    per-seeder budget separation test)
  - 47/47 across Sprint 1 + cohort follow-up stack
  - typecheck:api clean; lint clean
  - Neither seeder is in Dockerfile.relay (verified via grep) — no
    relay-COPY change needed

Sprint 4 is now done for the WB cohort (3 of 5 plan-listed indicators
migrated, with a 4th — IMF/WEO — explicitly deferred because it has
forecast-year semantics that need different content-age handling).
…rder)

P2 — `tests/wb-country-dict-content-age.test.mjs:79` — misleading inline
comment: read `// end-of-2026 = Dec 31 23:59:59 = past FIXED_NOW (May 5)`
but FIXED_NOW is 2026-05-05 and end-of-2026 is ~7 months in the FUTURE,
not past. The test logic is correct (the EDGE year IS excluded as
future-dated beyond skew tolerance) — only the comment was wrong.

P2 — `scripts/seed-fossil-electricity-share.mjs:30` — `import iso3ToIso2`
appeared on the line immediately after `const MAX_CONTENT_AGE_MIN`.
ES module `import`s are hoisted regardless of source order, but
interleaving with declarations confuses readers (code "looks" sequential
but the import actually executes first). Moved the import up alongside
the other top-of-module imports.

Both pure-text nits — no behavior change. typecheck clean; targeted
tests/wb-country-dict-content-age.test.mjs passes 15/15.
@koala73
koala73 force-pushed the feat/health-readiness-content-age-sprint-4-cohort-fossil-low-carbon branch from 0355a36 to 716a396 Compare May 8, 2026 08:59
koala73 added a commit that referenced this pull request May 8, 2026
…tics, 18mo budget)

Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed
"plus IMF/WEO/etc." as part of the annual-data migration). Branched off
Sprint 1 (#3596) as a parallel sibling.

Migrates all 4 IMF SDMX seeders in one PR:
  - seed-imf-external.mjs   (BCA, TM_RPCH, TX_RPCH)
  - seed-imf-growth.mjs     (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP)
  - seed-imf-labor.mjs      (LUR, LP)
  - seed-imf-macro.mjs      (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP)

## The semantic difference from WB cohort (and why a separate helper)

WB indicators store the OBSERVED year — `record.date = "2024"` means
data observed during calendar year 2024. The WB helper maps year →
end-of-year UTC ms (the latest observation date inside the named year).

IMF/WEO stores the FORECAST horizon, NOT an observation year. The
`weoYears()` function in `_seed-utils.mjs` returns
`[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks
the first year that has a finite value. So in May 2026 after the April
2026 WEO release, max stored year = 2026 — that's IMF's freshest
*forecast* for fiscal 2026, not observations through end-of-2026.

If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`):
year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to
NOW → rejected by 1h skew limit → `contentMeta` returns null → every
fresh IMF cache reports STALE_CONTENT. That's the failure mode this
module avoids.

Mapping rationale: `imfForecastYearToMs(year)` returns
`Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest
fully-observed period this forecast vintage is built on." For year=2026
→ end-of-2025 = ~5 months ago in May 2026. Correctly fresh.

A dedicated test (`semantic difference from WB cohort: forecast year
2026 in May 2026 maps to past (NOT future)`) exists specifically to
prevent a future refactor from collapsing the WB and IMF helpers.

## Why one shared budget across all 4 IMF seeders (NOT per-seeder)

WB cohort had per-seeder budgets because publication lags differed
(LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL
upstream — IMF SDMX/WEO. WEO publishes April + October vintages each
year as a single integrated release covering all WorldMonitor's
indicator codes. So all 4 share the same fresh-arrival lag and the
same steady-state ceiling. One budget = correct.

## 18-month budget — derivation

Steady-state model under "year → end-of-(year-1)" mapping:

  - After April N release: max year = N → newestItemAt = end-of-(N-1).
    Age = ~5 months.
  - After October N: max year still = N → age = ~11 months.
  - Just before April N+1: max year still = N → age = ~16 months.
  - After April N+1: max year advances to N+1 → newestItemAt resets.

Steady-state ceiling = 16mo (just before April release of next year).
Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO
releases is missed (both April AND October vintages of one year), which
is the right pager threshold for an IMF outage.

## Verification

  - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady-
    state regression guards, future-skew defense, late-reporter cohort
    handling, and the WB-vs-IMF semantic-difference guard test)
  - Tested with `npx tsx --test` against the existing IMF test suites:
    34/34 across `imf-country-data` + `seed-imf-extended` + new file
  - 47/47 across Sprint 1 + IMF cohort stack
  - typecheck:api clean; lint clean
  - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep)
    so no relay-COPY change needed

## Sprint 4 status after this PR

  - ✅ power-reliability (#3602)
  - ✅ low-carbon-generation + fossil-electricity-share (#3603)
  - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR)

Plan §477-485 fully closed. The plan's "Definition of done" §530
(≥1 annual-data migrated) was satisfied by #3602; this PR + #3603
round out the rest of the listed cohort.
@koala73
koala73 merged commit 15149f1 into main May 8, 2026
10 checks passed
@koala73
koala73 deleted the feat/health-readiness-content-age-sprint-4-cohort-fossil-low-carbon branch May 8, 2026 09:07
koala73 added a commit that referenced this pull request May 8, 2026
…tics, 18mo budget) (#3604)

* feat(imf-weo): Sprint 4 IMF cohort — content-age (forecast-year semantics, 18mo budget)

Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed
"plus IMF/WEO/etc." as part of the annual-data migration). Branched off
Sprint 1 (#3596) as a parallel sibling.

Migrates all 4 IMF SDMX seeders in one PR:
  - seed-imf-external.mjs   (BCA, TM_RPCH, TX_RPCH)
  - seed-imf-growth.mjs     (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP)
  - seed-imf-labor.mjs      (LUR, LP)
  - seed-imf-macro.mjs      (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP)

## The semantic difference from WB cohort (and why a separate helper)

WB indicators store the OBSERVED year — `record.date = "2024"` means
data observed during calendar year 2024. The WB helper maps year →
end-of-year UTC ms (the latest observation date inside the named year).

IMF/WEO stores the FORECAST horizon, NOT an observation year. The
`weoYears()` function in `_seed-utils.mjs` returns
`[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks
the first year that has a finite value. So in May 2026 after the April
2026 WEO release, max stored year = 2026 — that's IMF's freshest
*forecast* for fiscal 2026, not observations through end-of-2026.

If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`):
year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to
NOW → rejected by 1h skew limit → `contentMeta` returns null → every
fresh IMF cache reports STALE_CONTENT. That's the failure mode this
module avoids.

Mapping rationale: `imfForecastYearToMs(year)` returns
`Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest
fully-observed period this forecast vintage is built on." For year=2026
→ end-of-2025 = ~5 months ago in May 2026. Correctly fresh.

A dedicated test (`semantic difference from WB cohort: forecast year
2026 in May 2026 maps to past (NOT future)`) exists specifically to
prevent a future refactor from collapsing the WB and IMF helpers.

## Why one shared budget across all 4 IMF seeders (NOT per-seeder)

WB cohort had per-seeder budgets because publication lags differed
(LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL
upstream — IMF SDMX/WEO. WEO publishes April + October vintages each
year as a single integrated release covering all WorldMonitor's
indicator codes. So all 4 share the same fresh-arrival lag and the
same steady-state ceiling. One budget = correct.

## 18-month budget — derivation

Steady-state model under "year → end-of-(year-1)" mapping:

  - After April N release: max year = N → newestItemAt = end-of-(N-1).
    Age = ~5 months.
  - After October N: max year still = N → age = ~11 months.
  - Just before April N+1: max year still = N → age = ~16 months.
  - After April N+1: max year advances to N+1 → newestItemAt resets.

Steady-state ceiling = 16mo (just before April release of next year).
Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO
releases is missed (both April AND October vintages of one year), which
is the right pager threshold for an IMF outage.

## Verification

  - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady-
    state regression guards, future-skew defense, late-reporter cohort
    handling, and the WB-vs-IMF semantic-difference guard test)
  - Tested with `npx tsx --test` against the existing IMF test suites:
    34/34 across `imf-country-data` + `seed-imf-extended` + new file
  - 47/47 across Sprint 1 + IMF cohort stack
  - typecheck:api clean; lint clean
  - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep)
    so no relay-COPY change needed

## Sprint 4 status after this PR

  - ✅ power-reliability (#3602)
  - ✅ low-carbon-generation + fossil-electricity-share (#3603)
  - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR)

Plan §477-485 fully closed. The plan's "Definition of done" §530
(≥1 annual-data migrated) was satisfied by #3602; this PR + #3603
round out the rest of the listed cohort.

* fix(imf-weo): use max forecast year for content-age, not priority-first metric

Codex PR #3604 P2. The four IMF/WEO seeders write `entry.year` as the
priority-first non-null indicator's year (`ca?.year ?? tm?.year ?? tx?.year`
in seed-imf-external). That's correct as the public payload's "primary
metric vintage" but WRONG for content-age: a row with BCA=2024 +
import-volume=2026 publishes year=2024, even though the country dict
carries a fresh 2026 metric — content-age maps it to 2023-12-31 (~17mo old,
near-stale) when it actually carries a 2026 metric (~5mo old in May 2026).

Fix path A (preserves public payload semantics): seeders now populate a
dedicated `latestYear` field via a new `maxIntegerYear()` helper, computed
across ALL the country's indicator years. The content-age helper prefers
`entry.latestYear` over `entry.year`, falling back to `year` for back-compat
with caches written before this PR.

- scripts/_imf-weo-content-age-helpers.mjs — export `maxIntegerYear()`;
  `imfWeoContentMeta` reads `entry.latestYear` first
- scripts/seed-imf-{external,growth,labor,macro}.mjs — populate `latestYear`
  alongside existing `year` (no public payload change beyond the new field)
- tests/imf-weo-content-age.test.mjs — add maxIntegerYear unit tests +
  three mixed-indicator-year regression tests covering the fresh-metric-
  behind-stale-primary case, latestYear=null fallback, and heterogeneous
  cohort newest/oldest extraction

* chore(imf-weo): adversarial-review hardening — horizon-extension trap guard + schemaVersion bump

PR #3604 review findings #1 + #2. Both advisory, no behavior change today.

#1 Horizon-extension trap: weoYears() currently returns [currentYear,
   currentYear-1, currentYear-2], so max year = currentYear and the 1h
   skew filter is purely defensive. If a future Sprint extends weoYears()
   to include currentYear+1 to surface forward forecasts, the skew filter
   would silently drop every fresh +1 entry, regressing cohort
   newestItemAt to the prior year and producing FALSE STALE_CONTENT for
   genuinely-fresh data. Added load-bearing comment near the skew check
   plus a regression-guard test that documents the trap shape under
   FIXED_NOW=2026-05-05. Test asserts the trap, not desired behavior;
   when horizon extension lands the test fails and forces revisit.

#2 schemaVersion bump 1->2 across all 4 seeders. Codex P2 added the
   latestYear field; envelope newestItemAt math now differs under the
   same schema number. Bumping forces a clean republish on rollout and
   makes rollback observable rather than silently drifting envelope math
   while caches keep the new shape.
fuleinist pushed a commit to fuleinist/worldmonitor that referenced this pull request May 9, 2026
…ssil-share (koala73#3603)

* feat(wb-cohort): Sprint 4 follow-up — content-age for low-carbon + fossil-share

Sprint 4 cohort follow-up of the 2026-05-04 health-readiness probe plan.
Migrates the two remaining WB resilience seeders that match power-reliability's
shape: seed-low-carbon-generation.mjs and seed-fossil-electricity-share.mjs.
Branched off Sprint 1 (koala73#3596) as a parallel sibling.

## Why a shared helper this time

Three production seeders now use the IDENTICAL per-country-dict shape
({countries: {ISO2: {value, year}}, seededAt}) with the IDENTICAL
contentMeta math (max-year selection + end-of-year UTC + 1h skew limit).
Per CLAUDE.md "three similar lines is better than a premature abstraction"
— three is exactly the line for justifying the abstraction now.

New `scripts/_wb-country-dict-content-age-helpers.mjs` exports:
  - yearToEndOfYearMs(year)
  - wbCountryDictContentMeta(data, nowMs?)

Each seeder imports it + brings its own MAX_CONTENT_AGE_MIN constant
inline (per-seeder budgets matter — see below). seed-power-reliability
keeps its own helper for now (PR koala73#3602 is in review; backporting to the
shared helper is a follow-up after merge to keep that PR's diff focused).
The math is verifiably identical.

## Per-seeder budgets (NOT one-size-fits-all)

Verified against live WB API on 2026-05-05 — publication lags differ
across these "annual WB indicators":

  - low-carbon-generation (NUCL+RNEW+HYRO sum, MAX year of 3):
      max year = 2024 (driven by NUCL/HYRO; RNEW lags to 2021 but is
      masked by MAX-of-3 in the seeder's countries[iso2].year compute)
      → fresh-arrival lag ~17mo
      → 36mo budget (= 30mo steady-state ceiling + 6mo slack)
      → matches power-reliability exactly

  - fossil-electricity-share (EG.ELC.FOSL.ZS):
      max year = 2023 (NOT 2024 — slower-publishing indicator)
      → fresh-arrival lag ~29mo
      → 48mo budget (= 41mo steady-state ceiling + 7mo slack)

A naive cohort-wide budget would either false-positive on fossil-share
(if 36mo) or be wastefully loose on low-carbon (if 48mo). Per-seeder
constants are the correct response — each indicator's lag is empirically
different.

The "per-seeder budget separation" test pins this explicitly: a 41mo cache
trips low-carbon (36mo) but NOT fossil-share (48mo). Demonstrates that the
budgets aren't accidental — they reflect real upstream cadence differences.

## Renewables (RNEW.ZS) data-quality flag

Discovered during the audit: EG.ELC.RNEW.ZS max year = 2021 in May 2026,
~53mo lag. Inside low-carbon-generation it's masked by MAX(NUCL, RNEW,
HYRO), so content-age looks fine. But the underlying renewable share
data is genuinely 5+ years stale. Not addressed in this PR — flagging
as a separate data-quality concern for follow-up review.

## Verification

  - 15/15 wb-country-dict content-age tests pass (incl. fresh-arrival +
    steady-state regression guards for BOTH new seeders, plus a
    per-seeder budget separation test)
  - 47/47 across Sprint 1 + cohort follow-up stack
  - typecheck:api clean; lint clean
  - Neither seeder is in Dockerfile.relay (verified via grep) — no
    relay-COPY change needed

Sprint 4 is now done for the WB cohort (3 of 5 plan-listed indicators
migrated, with a 4th — IMF/WEO — explicitly deferred because it has
forecast-year semantics that need different content-age handling).

* fix: address Greptile PR koala73#3603 P2 nits (misleading comment + import order)

P2 — `tests/wb-country-dict-content-age.test.mjs:79` — misleading inline
comment: read `// end-of-2026 = Dec 31 23:59:59 = past FIXED_NOW (May 5)`
but FIXED_NOW is 2026-05-05 and end-of-2026 is ~7 months in the FUTURE,
not past. The test logic is correct (the EDGE year IS excluded as
future-dated beyond skew tolerance) — only the comment was wrong.

P2 — `scripts/seed-fossil-electricity-share.mjs:30` — `import iso3ToIso2`
appeared on the line immediately after `const MAX_CONTENT_AGE_MIN`.
ES module `import`s are hoisted regardless of source order, but
interleaving with declarations confuses readers (code "looks" sequential
but the import actually executes first). Moved the import up alongside
the other top-of-module imports.

Both pure-text nits — no behavior change. typecheck clean; targeted
tests/wb-country-dict-content-age.test.mjs passes 15/15.
fuleinist pushed a commit to fuleinist/worldmonitor that referenced this pull request May 9, 2026
…tics, 18mo budget) (koala73#3604)

* feat(imf-weo): Sprint 4 IMF cohort — content-age (forecast-year semantics, 18mo budget)

Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed
"plus IMF/WEO/etc." as part of the annual-data migration). Branched off
Sprint 1 (koala73#3596) as a parallel sibling.

Migrates all 4 IMF SDMX seeders in one PR:
  - seed-imf-external.mjs   (BCA, TM_RPCH, TX_RPCH)
  - seed-imf-growth.mjs     (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP)
  - seed-imf-labor.mjs      (LUR, LP)
  - seed-imf-macro.mjs      (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP)

## The semantic difference from WB cohort (and why a separate helper)

WB indicators store the OBSERVED year — `record.date = "2024"` means
data observed during calendar year 2024. The WB helper maps year →
end-of-year UTC ms (the latest observation date inside the named year).

IMF/WEO stores the FORECAST horizon, NOT an observation year. The
`weoYears()` function in `_seed-utils.mjs` returns
`[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks
the first year that has a finite value. So in May 2026 after the April
2026 WEO release, max stored year = 2026 — that's IMF's freshest
*forecast* for fiscal 2026, not observations through end-of-2026.

If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`):
year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to
NOW → rejected by 1h skew limit → `contentMeta` returns null → every
fresh IMF cache reports STALE_CONTENT. That's the failure mode this
module avoids.

Mapping rationale: `imfForecastYearToMs(year)` returns
`Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest
fully-observed period this forecast vintage is built on." For year=2026
→ end-of-2025 = ~5 months ago in May 2026. Correctly fresh.

A dedicated test (`semantic difference from WB cohort: forecast year
2026 in May 2026 maps to past (NOT future)`) exists specifically to
prevent a future refactor from collapsing the WB and IMF helpers.

## Why one shared budget across all 4 IMF seeders (NOT per-seeder)

WB cohort had per-seeder budgets because publication lags differed
(LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL
upstream — IMF SDMX/WEO. WEO publishes April + October vintages each
year as a single integrated release covering all WorldMonitor's
indicator codes. So all 4 share the same fresh-arrival lag and the
same steady-state ceiling. One budget = correct.

## 18-month budget — derivation

Steady-state model under "year → end-of-(year-1)" mapping:

  - After April N release: max year = N → newestItemAt = end-of-(N-1).
    Age = ~5 months.
  - After October N: max year still = N → age = ~11 months.
  - Just before April N+1: max year still = N → age = ~16 months.
  - After April N+1: max year advances to N+1 → newestItemAt resets.

Steady-state ceiling = 16mo (just before April release of next year).
Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO
releases is missed (both April AND October vintages of one year), which
is the right pager threshold for an IMF outage.

## Verification

  - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady-
    state regression guards, future-skew defense, late-reporter cohort
    handling, and the WB-vs-IMF semantic-difference guard test)
  - Tested with `npx tsx --test` against the existing IMF test suites:
    34/34 across `imf-country-data` + `seed-imf-extended` + new file
  - 47/47 across Sprint 1 + IMF cohort stack
  - typecheck:api clean; lint clean
  - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep)
    so no relay-COPY change needed

## Sprint 4 status after this PR

  - ✅ power-reliability (koala73#3602)
  - ✅ low-carbon-generation + fossil-electricity-share (koala73#3603)
  - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR)

Plan §477-485 fully closed. The plan's "Definition of done" §530
(≥1 annual-data migrated) was satisfied by koala73#3602; this PR + koala73#3603
round out the rest of the listed cohort.

* fix(imf-weo): use max forecast year for content-age, not priority-first metric

Codex PR koala73#3604 P2. The four IMF/WEO seeders write `entry.year` as the
priority-first non-null indicator's year (`ca?.year ?? tm?.year ?? tx?.year`
in seed-imf-external). That's correct as the public payload's "primary
metric vintage" but WRONG for content-age: a row with BCA=2024 +
import-volume=2026 publishes year=2024, even though the country dict
carries a fresh 2026 metric — content-age maps it to 2023-12-31 (~17mo old,
near-stale) when it actually carries a 2026 metric (~5mo old in May 2026).

Fix path A (preserves public payload semantics): seeders now populate a
dedicated `latestYear` field via a new `maxIntegerYear()` helper, computed
across ALL the country's indicator years. The content-age helper prefers
`entry.latestYear` over `entry.year`, falling back to `year` for back-compat
with caches written before this PR.

- scripts/_imf-weo-content-age-helpers.mjs — export `maxIntegerYear()`;
  `imfWeoContentMeta` reads `entry.latestYear` first
- scripts/seed-imf-{external,growth,labor,macro}.mjs — populate `latestYear`
  alongside existing `year` (no public payload change beyond the new field)
- tests/imf-weo-content-age.test.mjs — add maxIntegerYear unit tests +
  three mixed-indicator-year regression tests covering the fresh-metric-
  behind-stale-primary case, latestYear=null fallback, and heterogeneous
  cohort newest/oldest extraction

* chore(imf-weo): adversarial-review hardening — horizon-extension trap guard + schemaVersion bump

PR koala73#3604 review findings #1 + #2. Both advisory, no behavior change today.

#1 Horizon-extension trap: weoYears() currently returns [currentYear,
   currentYear-1, currentYear-2], so max year = currentYear and the 1h
   skew filter is purely defensive. If a future Sprint extends weoYears()
   to include currentYear+1 to surface forward forecasts, the skew filter
   would silently drop every fresh +1 entry, regressing cohort
   newestItemAt to the prior year and producing FALSE STALE_CONTENT for
   genuinely-fresh data. Added load-bearing comment near the skew check
   plus a regression-guard test that documents the trap shape under
   FIXED_NOW=2026-05-05. Test asserts the trap, not desired behavior;
   when horizon extension lands the test fails and forces revisit.

#2 schemaVersion bump 1->2 across all 4 seeders. Codex P2 added the
   latestYear field; envelope newestItemAt math now differs under the
   same schema number. Bumping forces a clean republish on rollout and
   makes rollback observable rather than silently drifting envelope math
   while caches keep the new shape.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant