feat(imf-weo): Sprint 4 IMF cohort — content-age (forecast-year semantics, 18mo budget)#3604
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryWires content-age health monitoring into all four IMF SDMX/WEO seeders via a new shared helper module (
Confidence Score: 5/5Safe to merge; the helper and seeders are additive opt-ins with no changes to existing fetch, parse, or publish logic. The forecast-year to end-of-(year-1) math is correct and well-tested; the 1-hour skew guard is properly defensive; contentMeta is called with a defaulting Date.now() second parameter which is correct for production; the 18-month budget derivation is clearly documented and validated by the steady-state and catastrophic-stall tests. No existing code paths were modified. No files require special attention. The helper module is the logical core and its arithmetic and guard conditions check out. Important Files Changed
Reviews (1): Last reviewed commit: "feat(imf-weo): Sprint 4 IMF cohort — con..." | Re-trigger Greptile |
…st metric Codex PR #3604 P2. The four IMF/WEO seeders write `entry.year` as the priority-first non-null indicator's year (`ca?.year ?? tm?.year ?? tx?.year` in seed-imf-external). That's correct as the public payload's "primary metric vintage" but WRONG for content-age: a row with BCA=2024 + import-volume=2026 publishes year=2024, even though the country dict carries a fresh 2026 metric — content-age maps it to 2023-12-31 (~17mo old, near-stale) when it actually carries a 2026 metric (~5mo old in May 2026). Fix path A (preserves public payload semantics): seeders now populate a dedicated `latestYear` field via a new `maxIntegerYear()` helper, computed across ALL the country's indicator years. The content-age helper prefers `entry.latestYear` over `entry.year`, falling back to `year` for back-compat with caches written before this PR. - scripts/_imf-weo-content-age-helpers.mjs — export `maxIntegerYear()`; `imfWeoContentMeta` reads `entry.latestYear` first - scripts/seed-imf-{external,growth,labor,macro}.mjs — populate `latestYear` alongside existing `year` (no public payload change beyond the new field) - tests/imf-weo-content-age.test.mjs — add maxIntegerYear unit tests + three mixed-indicator-year regression tests covering the fresh-metric- behind-stale-primary case, latestYear=null fallback, and heterogeneous cohort newest/oldest extraction
… guard + schemaVersion bump PR #3604 review findings #1 + #2. Both advisory, no behavior change today. #1 Horizon-extension trap: weoYears() currently returns [currentYear, currentYear-1, currentYear-2], so max year = currentYear and the 1h skew filter is purely defensive. If a future Sprint extends weoYears() to include currentYear+1 to surface forward forecasts, the skew filter would silently drop every fresh +1 entry, regressing cohort newestItemAt to the prior year and producing FALSE STALE_CONTENT for genuinely-fresh data. Added load-bearing comment near the skew check plus a regression-guard test that documents the trap shape under FIXED_NOW=2026-05-05. Test asserts the trap, not desired behavior; when horizon extension lands the test fails and forces revisit. #2 schemaVersion bump 1->2 across all 4 seeders. Codex P2 added the latestYear field; envelope newestItemAt math now differs under the same schema number. Bumping forces a clean republish on rollout and makes rollback observable rather than silently drifting envelope math while caches keep the new shape.
…tics, 18mo budget) Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed "plus IMF/WEO/etc." as part of the annual-data migration). Branched off Sprint 1 (#3596) as a parallel sibling. Migrates all 4 IMF SDMX seeders in one PR: - seed-imf-external.mjs (BCA, TM_RPCH, TX_RPCH) - seed-imf-growth.mjs (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP) - seed-imf-labor.mjs (LUR, LP) - seed-imf-macro.mjs (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP) ## The semantic difference from WB cohort (and why a separate helper) WB indicators store the OBSERVED year — `record.date = "2024"` means data observed during calendar year 2024. The WB helper maps year → end-of-year UTC ms (the latest observation date inside the named year). IMF/WEO stores the FORECAST horizon, NOT an observation year. The `weoYears()` function in `_seed-utils.mjs` returns `[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks the first year that has a finite value. So in May 2026 after the April 2026 WEO release, max stored year = 2026 — that's IMF's freshest *forecast* for fiscal 2026, not observations through end-of-2026. If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`): year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to NOW → rejected by 1h skew limit → `contentMeta` returns null → every fresh IMF cache reports STALE_CONTENT. That's the failure mode this module avoids. Mapping rationale: `imfForecastYearToMs(year)` returns `Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest fully-observed period this forecast vintage is built on." For year=2026 → end-of-2025 = ~5 months ago in May 2026. Correctly fresh. A dedicated test (`semantic difference from WB cohort: forecast year 2026 in May 2026 maps to past (NOT future)`) exists specifically to prevent a future refactor from collapsing the WB and IMF helpers. ## Why one shared budget across all 4 IMF seeders (NOT per-seeder) WB cohort had per-seeder budgets because publication lags differed (LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL upstream — IMF SDMX/WEO. WEO publishes April + October vintages each year as a single integrated release covering all WorldMonitor's indicator codes. So all 4 share the same fresh-arrival lag and the same steady-state ceiling. One budget = correct. ## 18-month budget — derivation Steady-state model under "year → end-of-(year-1)" mapping: - After April N release: max year = N → newestItemAt = end-of-(N-1). Age = ~5 months. - After October N: max year still = N → age = ~11 months. - Just before April N+1: max year still = N → age = ~16 months. - After April N+1: max year advances to N+1 → newestItemAt resets. Steady-state ceiling = 16mo (just before April release of next year). Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO releases is missed (both April AND October vintages of one year), which is the right pager threshold for an IMF outage. ## Verification - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady- state regression guards, future-skew defense, late-reporter cohort handling, and the WB-vs-IMF semantic-difference guard test) - Tested with `npx tsx --test` against the existing IMF test suites: 34/34 across `imf-country-data` + `seed-imf-extended` + new file - 47/47 across Sprint 1 + IMF cohort stack - typecheck:api clean; lint clean - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep) so no relay-COPY change needed ## Sprint 4 status after this PR - ✅ power-reliability (#3602) - ✅ low-carbon-generation + fossil-electricity-share (#3603) - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR) Plan §477-485 fully closed. The plan's "Definition of done" §530 (≥1 annual-data migrated) was satisfied by #3602; this PR + #3603 round out the rest of the listed cohort.
…st metric Codex PR #3604 P2. The four IMF/WEO seeders write `entry.year` as the priority-first non-null indicator's year (`ca?.year ?? tm?.year ?? tx?.year` in seed-imf-external). That's correct as the public payload's "primary metric vintage" but WRONG for content-age: a row with BCA=2024 + import-volume=2026 publishes year=2024, even though the country dict carries a fresh 2026 metric — content-age maps it to 2023-12-31 (~17mo old, near-stale) when it actually carries a 2026 metric (~5mo old in May 2026). Fix path A (preserves public payload semantics): seeders now populate a dedicated `latestYear` field via a new `maxIntegerYear()` helper, computed across ALL the country's indicator years. The content-age helper prefers `entry.latestYear` over `entry.year`, falling back to `year` for back-compat with caches written before this PR. - scripts/_imf-weo-content-age-helpers.mjs — export `maxIntegerYear()`; `imfWeoContentMeta` reads `entry.latestYear` first - scripts/seed-imf-{external,growth,labor,macro}.mjs — populate `latestYear` alongside existing `year` (no public payload change beyond the new field) - tests/imf-weo-content-age.test.mjs — add maxIntegerYear unit tests + three mixed-indicator-year regression tests covering the fresh-metric- behind-stale-primary case, latestYear=null fallback, and heterogeneous cohort newest/oldest extraction
… guard + schemaVersion bump PR #3604 review findings #1 + #2. Both advisory, no behavior change today. #1 Horizon-extension trap: weoYears() currently returns [currentYear, currentYear-1, currentYear-2], so max year = currentYear and the 1h skew filter is purely defensive. If a future Sprint extends weoYears() to include currentYear+1 to surface forward forecasts, the skew filter would silently drop every fresh +1 entry, regressing cohort newestItemAt to the prior year and producing FALSE STALE_CONTENT for genuinely-fresh data. Added load-bearing comment near the skew check plus a regression-guard test that documents the trap shape under FIXED_NOW=2026-05-05. Test asserts the trap, not desired behavior; when horizon extension lands the test fails and forces revisit. #2 schemaVersion bump 1->2 across all 4 seeders. Codex P2 added the latestYear field; envelope newestItemAt math now differs under the same schema number. Bumping forces a clean republish on rollout and makes rollback observable rather than silently drifting envelope math while caches keep the new shape.
40d67eb to
11425f2
Compare
…tics, 18mo budget) (koala73#3604) * feat(imf-weo): Sprint 4 IMF cohort — content-age (forecast-year semantics, 18mo budget) Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed "plus IMF/WEO/etc." as part of the annual-data migration). Branched off Sprint 1 (koala73#3596) as a parallel sibling. Migrates all 4 IMF SDMX seeders in one PR: - seed-imf-external.mjs (BCA, TM_RPCH, TX_RPCH) - seed-imf-growth.mjs (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP) - seed-imf-labor.mjs (LUR, LP) - seed-imf-macro.mjs (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP) ## The semantic difference from WB cohort (and why a separate helper) WB indicators store the OBSERVED year — `record.date = "2024"` means data observed during calendar year 2024. The WB helper maps year → end-of-year UTC ms (the latest observation date inside the named year). IMF/WEO stores the FORECAST horizon, NOT an observation year. The `weoYears()` function in `_seed-utils.mjs` returns `[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks the first year that has a finite value. So in May 2026 after the April 2026 WEO release, max stored year = 2026 — that's IMF's freshest *forecast* for fiscal 2026, not observations through end-of-2026. If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`): year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to NOW → rejected by 1h skew limit → `contentMeta` returns null → every fresh IMF cache reports STALE_CONTENT. That's the failure mode this module avoids. Mapping rationale: `imfForecastYearToMs(year)` returns `Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest fully-observed period this forecast vintage is built on." For year=2026 → end-of-2025 = ~5 months ago in May 2026. Correctly fresh. A dedicated test (`semantic difference from WB cohort: forecast year 2026 in May 2026 maps to past (NOT future)`) exists specifically to prevent a future refactor from collapsing the WB and IMF helpers. ## Why one shared budget across all 4 IMF seeders (NOT per-seeder) WB cohort had per-seeder budgets because publication lags differed (LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL upstream — IMF SDMX/WEO. WEO publishes April + October vintages each year as a single integrated release covering all WorldMonitor's indicator codes. So all 4 share the same fresh-arrival lag and the same steady-state ceiling. One budget = correct. ## 18-month budget — derivation Steady-state model under "year → end-of-(year-1)" mapping: - After April N release: max year = N → newestItemAt = end-of-(N-1). Age = ~5 months. - After October N: max year still = N → age = ~11 months. - Just before April N+1: max year still = N → age = ~16 months. - After April N+1: max year advances to N+1 → newestItemAt resets. Steady-state ceiling = 16mo (just before April release of next year). Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO releases is missed (both April AND October vintages of one year), which is the right pager threshold for an IMF outage. ## Verification - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady- state regression guards, future-skew defense, late-reporter cohort handling, and the WB-vs-IMF semantic-difference guard test) - Tested with `npx tsx --test` against the existing IMF test suites: 34/34 across `imf-country-data` + `seed-imf-extended` + new file - 47/47 across Sprint 1 + IMF cohort stack - typecheck:api clean; lint clean - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep) so no relay-COPY change needed ## Sprint 4 status after this PR - ✅ power-reliability (koala73#3602) - ✅ low-carbon-generation + fossil-electricity-share (koala73#3603) - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR) Plan §477-485 fully closed. The plan's "Definition of done" §530 (≥1 annual-data migrated) was satisfied by koala73#3602; this PR + koala73#3603 round out the rest of the listed cohort. * fix(imf-weo): use max forecast year for content-age, not priority-first metric Codex PR koala73#3604 P2. The four IMF/WEO seeders write `entry.year` as the priority-first non-null indicator's year (`ca?.year ?? tm?.year ?? tx?.year` in seed-imf-external). That's correct as the public payload's "primary metric vintage" but WRONG for content-age: a row with BCA=2024 + import-volume=2026 publishes year=2024, even though the country dict carries a fresh 2026 metric — content-age maps it to 2023-12-31 (~17mo old, near-stale) when it actually carries a 2026 metric (~5mo old in May 2026). Fix path A (preserves public payload semantics): seeders now populate a dedicated `latestYear` field via a new `maxIntegerYear()` helper, computed across ALL the country's indicator years. The content-age helper prefers `entry.latestYear` over `entry.year`, falling back to `year` for back-compat with caches written before this PR. - scripts/_imf-weo-content-age-helpers.mjs — export `maxIntegerYear()`; `imfWeoContentMeta` reads `entry.latestYear` first - scripts/seed-imf-{external,growth,labor,macro}.mjs — populate `latestYear` alongside existing `year` (no public payload change beyond the new field) - tests/imf-weo-content-age.test.mjs — add maxIntegerYear unit tests + three mixed-indicator-year regression tests covering the fresh-metric- behind-stale-primary case, latestYear=null fallback, and heterogeneous cohort newest/oldest extraction * chore(imf-weo): adversarial-review hardening — horizon-extension trap guard + schemaVersion bump PR koala73#3604 review findings #1 + #2. Both advisory, no behavior change today. #1 Horizon-extension trap: weoYears() currently returns [currentYear, currentYear-1, currentYear-2], so max year = currentYear and the 1h skew filter is purely defensive. If a future Sprint extends weoYears() to include currentYear+1 to surface forward forecasts, the skew filter would silently drop every fresh +1 entry, regressing cohort newestItemAt to the prior year and producing FALSE STALE_CONTENT for genuinely-fresh data. Added load-bearing comment near the skew check plus a regression-guard test that documents the trap shape under FIXED_NOW=2026-05-05. Test asserts the trap, not desired behavior; when horizon extension lands the test fails and forces revisit. #2 schemaVersion bump 1->2 across all 4 seeders. Codex P2 added the latestYear field; envelope newestItemAt math now differs under the same schema number. Bumping forces a clean republish on rollout and makes rollback observable rather than silently drifting envelope math while caches keep the new shape.
Closes the deferred IMF/WEO portion of Sprint 4 of the 2026-05-04 health-readiness probe plan. Plan §477-485 listed "plus IMF/WEO/etc." as part of the annual-data migration. Branched off Sprint 1 (#3596) as a parallel sibling.
Summary
Migrates all 4 IMF SDMX seeders in one PR (they share the same upstream and same publication cadence):
New helper `scripts/_imf-weo-content-age-helpers.mjs`: `imfForecastYearToMs`, `imfWeoContentMeta` (with injectable `nowMs`), `IMF_WEO_MAX_CONTENT_AGE_MIN`. All 4 seeders import it with the same constant.
Why a separate helper from the WB cohort
WB and IMF have superficially identical `{countries: {ISO2: {year}}}` shapes but opposite `year` semantics:
The IMF helper maps year → end-of-(year - 1) UTC ms. Reads as: "the latest fully-observed period this forecast vintage is built on." For year=2026 → end-of-2025 = ~5mo ago in May 2026 → correctly fresh.
A dedicated test (`semantic difference from WB cohort: forecast year 2026 in May 2026 maps to past (NOT future)`) exists specifically to prevent a future refactor from accidentally collapsing the WB and IMF helpers.
Why one shared budget across all 4 IMF seeders (NOT per-seeder)
The WB cohort needed per-seeder budgets because publication lags differed (LOSS at ~17mo, FOSL at ~29mo). But all 4 IMF seeders fetch from the IDENTICAL upstream — IMF SDMX/WEO. WEO publishes April + October vintages each year as a single integrated release covering all WorldMonitor's indicator codes. So all 4 share the same fresh-arrival lag and steady-state ceiling. One budget = correct.
18-month budget — derivation
Steady-state model under "year → end-of-(year-1)" mapping:
Budget = 16mo ceiling + 2mo slack = 18 months. Trips when a full year of WEO releases is missed (both April AND October vintages of one year), which is the right pager threshold for an IMF outage.
Test plan
Sprint 4 status after this PR
Plan §477-485 fully closed. The plan's "Definition of done" §530 (≥1 annual-data migrated) was satisfied by #3602; this PR + #3603 round out the rest of the listed cohort.