feat(power-reliability): Sprint 4 — content-age probe (24-month budget)#3602
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryAdds the content-age contract to the power-reliability seeder (
Confidence Score: 4/5Safe to merge; the staleness budget and helper logic are well-reasoned and the test suite directly guards the known false-positive failure mode from Sprint 3b. The change is narrowly scoped — a new pure-helper module, two extra lines in the seeder's options object, and a focused test file. The core logic (max-year selection, end-of-year UTC conversion, future-year guard) is straightforward and covered by 14 tests. The one rough edge is that the JSDoc comment on the budget constant describes it as "730 days" while the expression evaluates to 720 days; the discrepancy is harmless given the generous slack in the budget but could mislead a future auditor. scripts/_power-reliability-helpers.mjs — the "730 days" comment in the JSDoc for POWER_RELIABILITY_MAX_CONTENT_AGE_MIN should be reconciled with the 720-day value the expression actually computes. Important Files Changed
Reviews (1): Last reviewed commit: "feat(power-reliability): Sprint 4 — cont..." | Re-trigger Greptile |
…eiling Greptile P1 on PR #3602: 24-month budget false-positives mid-cycle when next-year data publishes legitimately late. The math I missed in the initial commit: fresh-arrival lag (~17mo for WB EG.ELC.LOSS.ZS) is the FLOOR but not the worst case. Once year N is in cache, it stays there until year N+1 publishes — which can legitimately take up to end-of-(N+1) + 18mo = end-of-N + 30mo under the documented 12-18 month publication-lag range. So cache age can reach 30 months between publications WITHOUT any real upstream stall. Corrected budget = 30mo steady-state ceiling + 6mo slack = 36 months (36 thirty-day-months = 1080 days ≈ 3 years). Also resolves the P2 prose-vs-math mismatch (JSDoc previously said "730 days" but `24 * 30 * 24 * 60` = 720; new wording "36 thirty-day months ≈ 1080 days" is internally consistent). General formula now documented in the helper JSDoc: budget >= max_publication_lag + cycle_length + slack Both halves required: fresh-arrival lag AND cycle_length. Initial PR covered fresh-arrival (~17mo) but missed cycle_length (12mo), which is exactly how the false-positive emerges. Same shape as Sprint 3b PR #3599 P1 — that one missed fresh-arrival; this one missed steady-state. Tests: - Renamed boundary test "max year 2023 (~29mo) DOES trip" → "steady-state regression guard: max year 2023 (~29mo) does NOT trip — within ceiling" with assertion direction flipped (29mo < 30mo ceiling = legitimate late-publication wait, not staleness) - Added new boundary test "max year 2022 (~40mo) DOES trip — past ceiling = real stall" to confirm the budget fires correctly past the ceiling - Constant assertion: 36 * 30 * 24 * 60 15/15 power-reliability tests pass; 47/47 across Sprint 1+4 stack; typecheck:api clean; lint clean.
Closes the plan's "Definition of done" item: at least 1 annual-data seeder migrated. Branched off Sprint 1 (#3596) as a parallel sibling to Sprints 2/3a/3b. ## Why this matters WB EG.ELC.LOSS.ZS publishes annually. Without a content-age probe, a stalled WB publication cycle is invisible to /api/health: the seeder runs fine on its 35-day TTL, fetchedAt stays fresh, but no country's year ever advances past e.g. 2024. STALE_CONTENT trips correctly when the cache stops advancing — for power-reliability, that means "by the time you'd expect year-N+1 data, year-N is still latest" → page on-call. ## Why 24 months (NOT the plan's 13 months) Plan §477-485 originally proposed `13 * 30 * 24 * 60` minutes (~13 months), but this is structurally wrong for WB indicators — verified against live WB API on 2026-05-05: curl https://api.worldbank.org/v2/country/USA;CHN;...;KWT/indicator/EG.ELC.LOSS.ZS On that date G7 max year = 2024. End-of-2024 = Dec 31 2024 = ~17 months before the seed. WB year-N data lands in cache 12-18 months after end-of-N (publication lag varies). A 13-month budget would have tripped STALE_CONTENT immediately on every successful fresh-arrival — the same failure mode Greptile P1 caught on Sprint 3b PR #3599 (45d budget vs M+2 60-day natural lag). 24mo math: - Year N data lands at age = 12-18 months (publication lag) - Year (N+1) data lands ~12 months later, resetting the clock - Worst case during steady state: age = ~30 months (just before next year drops AND publication lag at upper end) - 24mo budget catches catastrophic stalls (>2y silent upstream) without false-positive paging during normal between-publications ## Shape contract — third distinct shape this sprint Per-country dict where each country has its OWN year (different from Sprint 2/3a per-item arrays AND from Sprint 3b single-snapshot period): {countries: {US: {value, year: 2024}, KW: {year: 2021}, ...}, seededAt} `newestItemAt` = end-of-(max year across all countries) — drives staleness. Late reporters (KW/QA/AE) lagging G7 don't drag the panel into STALE_CONTENT; once any country's year advances, the clock resets. `oldestItemAt` = end-of-(min year across countries) — informational. ## Pattern parity with Sprint 2/3a/3b Pure helpers in `scripts/_power-reliability-helpers.mjs`: `yearToEndOfYearMs`, `powerReliabilityContentMeta` (with injectable `nowMs`), `POWER_RELIABILITY_MAX_CONTENT_AGE_MIN`. Seeder imports; test imports. No replicas. `seed-power-reliability.mjs` is NOT in Dockerfile.relay (verified via grep), so no COPY-line update needed. ## Verification - 14/14 power-reliability content-age tests pass - 46/46 across Sprint 1 + Sprint 4 stack - typecheck:api clean; lint clean - Tests include a dedicated `fresh-arrival regression guard` test that pins the EXACT budget/natural-lag mismatch failure mode (Sprint 3b lesson made concrete) so a future budget tightening cannot silently re-introduce the immediate-page bug - Boundary test: 2023 data in May 2026 (~29mo) DOES trip — confirms the staleness clock works correctly past the budget threshold
…eiling Greptile P1 on PR #3602: 24-month budget false-positives mid-cycle when next-year data publishes legitimately late. The math I missed in the initial commit: fresh-arrival lag (~17mo for WB EG.ELC.LOSS.ZS) is the FLOOR but not the worst case. Once year N is in cache, it stays there until year N+1 publishes — which can legitimately take up to end-of-(N+1) + 18mo = end-of-N + 30mo under the documented 12-18 month publication-lag range. So cache age can reach 30 months between publications WITHOUT any real upstream stall. Corrected budget = 30mo steady-state ceiling + 6mo slack = 36 months (36 thirty-day-months = 1080 days ≈ 3 years). Also resolves the P2 prose-vs-math mismatch (JSDoc previously said "730 days" but `24 * 30 * 24 * 60` = 720; new wording "36 thirty-day months ≈ 1080 days" is internally consistent). General formula now documented in the helper JSDoc: budget >= max_publication_lag + cycle_length + slack Both halves required: fresh-arrival lag AND cycle_length. Initial PR covered fresh-arrival (~17mo) but missed cycle_length (12mo), which is exactly how the false-positive emerges. Same shape as Sprint 3b PR #3599 P1 — that one missed fresh-arrival; this one missed steady-state. Tests: - Renamed boundary test "max year 2023 (~29mo) DOES trip" → "steady-state regression guard: max year 2023 (~29mo) does NOT trip — within ceiling" with assertion direction flipped (29mo < 30mo ceiling = legitimate late-publication wait, not staleness) - Added new boundary test "max year 2022 (~40mo) DOES trip — past ceiling = real stall" to confirm the budget fires correctly past the ceiling - Constant assertion: 36 * 30 * 24 * 60 15/15 power-reliability tests pass; 47/47 across Sprint 1+4 stack; typecheck:api clean; lint clean.
1764752 to
c7b6ea4
Compare
…ssil-share Sprint 4 cohort follow-up of the 2026-05-04 health-readiness probe plan. Migrates the two remaining WB resilience seeders that match power-reliability's shape: seed-low-carbon-generation.mjs and seed-fossil-electricity-share.mjs. Branched off Sprint 1 (#3596) as a parallel sibling. ## Why a shared helper this time Three production seeders now use the IDENTICAL per-country-dict shape ({countries: {ISO2: {value, year}}, seededAt}) with the IDENTICAL contentMeta math (max-year selection + end-of-year UTC + 1h skew limit). Per CLAUDE.md "three similar lines is better than a premature abstraction" — three is exactly the line for justifying the abstraction now. New `scripts/_wb-country-dict-content-age-helpers.mjs` exports: - yearToEndOfYearMs(year) - wbCountryDictContentMeta(data, nowMs?) Each seeder imports it + brings its own MAX_CONTENT_AGE_MIN constant inline (per-seeder budgets matter — see below). seed-power-reliability keeps its own helper for now (PR #3602 is in review; backporting to the shared helper is a follow-up after merge to keep that PR's diff focused). The math is verifiably identical. ## Per-seeder budgets (NOT one-size-fits-all) Verified against live WB API on 2026-05-05 — publication lags differ across these "annual WB indicators": - low-carbon-generation (NUCL+RNEW+HYRO sum, MAX year of 3): max year = 2024 (driven by NUCL/HYRO; RNEW lags to 2021 but is masked by MAX-of-3 in the seeder's countries[iso2].year compute) → fresh-arrival lag ~17mo → 36mo budget (= 30mo steady-state ceiling + 6mo slack) → matches power-reliability exactly - fossil-electricity-share (EG.ELC.FOSL.ZS): max year = 2023 (NOT 2024 — slower-publishing indicator) → fresh-arrival lag ~29mo → 48mo budget (= 41mo steady-state ceiling + 7mo slack) A naive cohort-wide budget would either false-positive on fossil-share (if 36mo) or be wastefully loose on low-carbon (if 48mo). Per-seeder constants are the correct response — each indicator's lag is empirically different. The "per-seeder budget separation" test pins this explicitly: a 41mo cache trips low-carbon (36mo) but NOT fossil-share (48mo). Demonstrates that the budgets aren't accidental — they reflect real upstream cadence differences. ## Renewables (RNEW.ZS) data-quality flag Discovered during the audit: EG.ELC.RNEW.ZS max year = 2021 in May 2026, ~53mo lag. Inside low-carbon-generation it's masked by MAX(NUCL, RNEW, HYRO), so content-age looks fine. But the underlying renewable share data is genuinely 5+ years stale. Not addressed in this PR — flagging as a separate data-quality concern for follow-up review. ## Verification - 15/15 wb-country-dict content-age tests pass (incl. fresh-arrival + steady-state regression guards for BOTH new seeders, plus a per-seeder budget separation test) - 47/47 across Sprint 1 + cohort follow-up stack - typecheck:api clean; lint clean - Neither seeder is in Dockerfile.relay (verified via grep) — no relay-COPY change needed Sprint 4 is now done for the WB cohort (3 of 5 plan-listed indicators migrated, with a 4th — IMF/WEO — explicitly deferred because it has forecast-year semantics that need different content-age handling).
…tics, 18mo budget) Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed "plus IMF/WEO/etc." as part of the annual-data migration). Branched off Sprint 1 (#3596) as a parallel sibling. Migrates all 4 IMF SDMX seeders in one PR: - seed-imf-external.mjs (BCA, TM_RPCH, TX_RPCH) - seed-imf-growth.mjs (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP) - seed-imf-labor.mjs (LUR, LP) - seed-imf-macro.mjs (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP) ## The semantic difference from WB cohort (and why a separate helper) WB indicators store the OBSERVED year — `record.date = "2024"` means data observed during calendar year 2024. The WB helper maps year → end-of-year UTC ms (the latest observation date inside the named year). IMF/WEO stores the FORECAST horizon, NOT an observation year. The `weoYears()` function in `_seed-utils.mjs` returns `[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks the first year that has a finite value. So in May 2026 after the April 2026 WEO release, max stored year = 2026 — that's IMF's freshest *forecast* for fiscal 2026, not observations through end-of-2026. If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`): year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to NOW → rejected by 1h skew limit → `contentMeta` returns null → every fresh IMF cache reports STALE_CONTENT. That's the failure mode this module avoids. Mapping rationale: `imfForecastYearToMs(year)` returns `Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest fully-observed period this forecast vintage is built on." For year=2026 → end-of-2025 = ~5 months ago in May 2026. Correctly fresh. A dedicated test (`semantic difference from WB cohort: forecast year 2026 in May 2026 maps to past (NOT future)`) exists specifically to prevent a future refactor from collapsing the WB and IMF helpers. ## Why one shared budget across all 4 IMF seeders (NOT per-seeder) WB cohort had per-seeder budgets because publication lags differed (LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL upstream — IMF SDMX/WEO. WEO publishes April + October vintages each year as a single integrated release covering all WorldMonitor's indicator codes. So all 4 share the same fresh-arrival lag and the same steady-state ceiling. One budget = correct. ## 18-month budget — derivation Steady-state model under "year → end-of-(year-1)" mapping: - After April N release: max year = N → newestItemAt = end-of-(N-1). Age = ~5 months. - After October N: max year still = N → age = ~11 months. - Just before April N+1: max year still = N → age = ~16 months. - After April N+1: max year advances to N+1 → newestItemAt resets. Steady-state ceiling = 16mo (just before April release of next year). Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO releases is missed (both April AND October vintages of one year), which is the right pager threshold for an IMF outage. ## Verification - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady- state regression guards, future-skew defense, late-reporter cohort handling, and the WB-vs-IMF semantic-difference guard test) - Tested with `npx tsx --test` against the existing IMF test suites: 34/34 across `imf-country-data` + `seed-imf-extended` + new file - 47/47 across Sprint 1 + IMF cohort stack - typecheck:api clean; lint clean - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep) so no relay-COPY change needed ## Sprint 4 status after this PR - ✅ power-reliability (#3602) - ✅ low-carbon-generation + fossil-electricity-share (#3603) - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR) Plan §477-485 fully closed. The plan's "Definition of done" §530 (≥1 annual-data migrated) was satisfied by #3602; this PR + #3603 round out the rest of the listed cohort.
…ssil-share (#3603) * feat(wb-cohort): Sprint 4 follow-up — content-age for low-carbon + fossil-share Sprint 4 cohort follow-up of the 2026-05-04 health-readiness probe plan. Migrates the two remaining WB resilience seeders that match power-reliability's shape: seed-low-carbon-generation.mjs and seed-fossil-electricity-share.mjs. Branched off Sprint 1 (#3596) as a parallel sibling. ## Why a shared helper this time Three production seeders now use the IDENTICAL per-country-dict shape ({countries: {ISO2: {value, year}}, seededAt}) with the IDENTICAL contentMeta math (max-year selection + end-of-year UTC + 1h skew limit). Per CLAUDE.md "three similar lines is better than a premature abstraction" — three is exactly the line for justifying the abstraction now. New `scripts/_wb-country-dict-content-age-helpers.mjs` exports: - yearToEndOfYearMs(year) - wbCountryDictContentMeta(data, nowMs?) Each seeder imports it + brings its own MAX_CONTENT_AGE_MIN constant inline (per-seeder budgets matter — see below). seed-power-reliability keeps its own helper for now (PR #3602 is in review; backporting to the shared helper is a follow-up after merge to keep that PR's diff focused). The math is verifiably identical. ## Per-seeder budgets (NOT one-size-fits-all) Verified against live WB API on 2026-05-05 — publication lags differ across these "annual WB indicators": - low-carbon-generation (NUCL+RNEW+HYRO sum, MAX year of 3): max year = 2024 (driven by NUCL/HYRO; RNEW lags to 2021 but is masked by MAX-of-3 in the seeder's countries[iso2].year compute) → fresh-arrival lag ~17mo → 36mo budget (= 30mo steady-state ceiling + 6mo slack) → matches power-reliability exactly - fossil-electricity-share (EG.ELC.FOSL.ZS): max year = 2023 (NOT 2024 — slower-publishing indicator) → fresh-arrival lag ~29mo → 48mo budget (= 41mo steady-state ceiling + 7mo slack) A naive cohort-wide budget would either false-positive on fossil-share (if 36mo) or be wastefully loose on low-carbon (if 48mo). Per-seeder constants are the correct response — each indicator's lag is empirically different. The "per-seeder budget separation" test pins this explicitly: a 41mo cache trips low-carbon (36mo) but NOT fossil-share (48mo). Demonstrates that the budgets aren't accidental — they reflect real upstream cadence differences. ## Renewables (RNEW.ZS) data-quality flag Discovered during the audit: EG.ELC.RNEW.ZS max year = 2021 in May 2026, ~53mo lag. Inside low-carbon-generation it's masked by MAX(NUCL, RNEW, HYRO), so content-age looks fine. But the underlying renewable share data is genuinely 5+ years stale. Not addressed in this PR — flagging as a separate data-quality concern for follow-up review. ## Verification - 15/15 wb-country-dict content-age tests pass (incl. fresh-arrival + steady-state regression guards for BOTH new seeders, plus a per-seeder budget separation test) - 47/47 across Sprint 1 + cohort follow-up stack - typecheck:api clean; lint clean - Neither seeder is in Dockerfile.relay (verified via grep) — no relay-COPY change needed Sprint 4 is now done for the WB cohort (3 of 5 plan-listed indicators migrated, with a 4th — IMF/WEO — explicitly deferred because it has forecast-year semantics that need different content-age handling). * fix: address Greptile PR #3603 P2 nits (misleading comment + import order) P2 — `tests/wb-country-dict-content-age.test.mjs:79` — misleading inline comment: read `// end-of-2026 = Dec 31 23:59:59 = past FIXED_NOW (May 5)` but FIXED_NOW is 2026-05-05 and end-of-2026 is ~7 months in the FUTURE, not past. The test logic is correct (the EDGE year IS excluded as future-dated beyond skew tolerance) — only the comment was wrong. P2 — `scripts/seed-fossil-electricity-share.mjs:30` — `import iso3ToIso2` appeared on the line immediately after `const MAX_CONTENT_AGE_MIN`. ES module `import`s are hoisted regardless of source order, but interleaving with declarations confuses readers (code "looks" sequential but the import actually executes first). Moved the import up alongside the other top-of-module imports. Both pure-text nits — no behavior change. typecheck clean; targeted tests/wb-country-dict-content-age.test.mjs passes 15/15.
…tics, 18mo budget) (#3604) * feat(imf-weo): Sprint 4 IMF cohort — content-age (forecast-year semantics, 18mo budget) Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed "plus IMF/WEO/etc." as part of the annual-data migration). Branched off Sprint 1 (#3596) as a parallel sibling. Migrates all 4 IMF SDMX seeders in one PR: - seed-imf-external.mjs (BCA, TM_RPCH, TX_RPCH) - seed-imf-growth.mjs (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP) - seed-imf-labor.mjs (LUR, LP) - seed-imf-macro.mjs (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP) ## The semantic difference from WB cohort (and why a separate helper) WB indicators store the OBSERVED year — `record.date = "2024"` means data observed during calendar year 2024. The WB helper maps year → end-of-year UTC ms (the latest observation date inside the named year). IMF/WEO stores the FORECAST horizon, NOT an observation year. The `weoYears()` function in `_seed-utils.mjs` returns `[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks the first year that has a finite value. So in May 2026 after the April 2026 WEO release, max stored year = 2026 — that's IMF's freshest *forecast* for fiscal 2026, not observations through end-of-2026. If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`): year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to NOW → rejected by 1h skew limit → `contentMeta` returns null → every fresh IMF cache reports STALE_CONTENT. That's the failure mode this module avoids. Mapping rationale: `imfForecastYearToMs(year)` returns `Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest fully-observed period this forecast vintage is built on." For year=2026 → end-of-2025 = ~5 months ago in May 2026. Correctly fresh. A dedicated test (`semantic difference from WB cohort: forecast year 2026 in May 2026 maps to past (NOT future)`) exists specifically to prevent a future refactor from collapsing the WB and IMF helpers. ## Why one shared budget across all 4 IMF seeders (NOT per-seeder) WB cohort had per-seeder budgets because publication lags differed (LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL upstream — IMF SDMX/WEO. WEO publishes April + October vintages each year as a single integrated release covering all WorldMonitor's indicator codes. So all 4 share the same fresh-arrival lag and the same steady-state ceiling. One budget = correct. ## 18-month budget — derivation Steady-state model under "year → end-of-(year-1)" mapping: - After April N release: max year = N → newestItemAt = end-of-(N-1). Age = ~5 months. - After October N: max year still = N → age = ~11 months. - Just before April N+1: max year still = N → age = ~16 months. - After April N+1: max year advances to N+1 → newestItemAt resets. Steady-state ceiling = 16mo (just before April release of next year). Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO releases is missed (both April AND October vintages of one year), which is the right pager threshold for an IMF outage. ## Verification - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady- state regression guards, future-skew defense, late-reporter cohort handling, and the WB-vs-IMF semantic-difference guard test) - Tested with `npx tsx --test` against the existing IMF test suites: 34/34 across `imf-country-data` + `seed-imf-extended` + new file - 47/47 across Sprint 1 + IMF cohort stack - typecheck:api clean; lint clean - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep) so no relay-COPY change needed ## Sprint 4 status after this PR - ✅ power-reliability (#3602) - ✅ low-carbon-generation + fossil-electricity-share (#3603) - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR) Plan §477-485 fully closed. The plan's "Definition of done" §530 (≥1 annual-data migrated) was satisfied by #3602; this PR + #3603 round out the rest of the listed cohort. * fix(imf-weo): use max forecast year for content-age, not priority-first metric Codex PR #3604 P2. The four IMF/WEO seeders write `entry.year` as the priority-first non-null indicator's year (`ca?.year ?? tm?.year ?? tx?.year` in seed-imf-external). That's correct as the public payload's "primary metric vintage" but WRONG for content-age: a row with BCA=2024 + import-volume=2026 publishes year=2024, even though the country dict carries a fresh 2026 metric — content-age maps it to 2023-12-31 (~17mo old, near-stale) when it actually carries a 2026 metric (~5mo old in May 2026). Fix path A (preserves public payload semantics): seeders now populate a dedicated `latestYear` field via a new `maxIntegerYear()` helper, computed across ALL the country's indicator years. The content-age helper prefers `entry.latestYear` over `entry.year`, falling back to `year` for back-compat with caches written before this PR. - scripts/_imf-weo-content-age-helpers.mjs — export `maxIntegerYear()`; `imfWeoContentMeta` reads `entry.latestYear` first - scripts/seed-imf-{external,growth,labor,macro}.mjs — populate `latestYear` alongside existing `year` (no public payload change beyond the new field) - tests/imf-weo-content-age.test.mjs — add maxIntegerYear unit tests + three mixed-indicator-year regression tests covering the fresh-metric- behind-stale-primary case, latestYear=null fallback, and heterogeneous cohort newest/oldest extraction * chore(imf-weo): adversarial-review hardening — horizon-extension trap guard + schemaVersion bump PR #3604 review findings #1 + #2. Both advisory, no behavior change today. #1 Horizon-extension trap: weoYears() currently returns [currentYear, currentYear-1, currentYear-2], so max year = currentYear and the 1h skew filter is purely defensive. If a future Sprint extends weoYears() to include currentYear+1 to surface forward forecasts, the skew filter would silently drop every fresh +1 entry, regressing cohort newestItemAt to the prior year and producing FALSE STALE_CONTENT for genuinely-fresh data. Added load-bearing comment near the skew check plus a regression-guard test that documents the trap shape under FIXED_NOW=2026-05-05. Test asserts the trap, not desired behavior; when horizon extension lands the test fails and forces revisit. #2 schemaVersion bump 1->2 across all 4 seeders. Codex P2 added the latestYear field; envelope newestItemAt math now differs under the same schema number. Bumping forces a clean republish on rollout and makes rollback observable rather than silently drifting envelope math while caches keep the new shape.
…t) (koala73#3602) * feat(power-reliability): Sprint 4 — content-age probe (24-month budget) Closes the plan's "Definition of done" item: at least 1 annual-data seeder migrated. Branched off Sprint 1 (koala73#3596) as a parallel sibling to Sprints 2/3a/3b. ## Why this matters WB EG.ELC.LOSS.ZS publishes annually. Without a content-age probe, a stalled WB publication cycle is invisible to /api/health: the seeder runs fine on its 35-day TTL, fetchedAt stays fresh, but no country's year ever advances past e.g. 2024. STALE_CONTENT trips correctly when the cache stops advancing — for power-reliability, that means "by the time you'd expect year-N+1 data, year-N is still latest" → page on-call. ## Why 24 months (NOT the plan's 13 months) Plan §477-485 originally proposed `13 * 30 * 24 * 60` minutes (~13 months), but this is structurally wrong for WB indicators — verified against live WB API on 2026-05-05: curl https://api.worldbank.org/v2/country/USA;CHN;...;KWT/indicator/EG.ELC.LOSS.ZS On that date G7 max year = 2024. End-of-2024 = Dec 31 2024 = ~17 months before the seed. WB year-N data lands in cache 12-18 months after end-of-N (publication lag varies). A 13-month budget would have tripped STALE_CONTENT immediately on every successful fresh-arrival — the same failure mode Greptile P1 caught on Sprint 3b PR koala73#3599 (45d budget vs M+2 60-day natural lag). 24mo math: - Year N data lands at age = 12-18 months (publication lag) - Year (N+1) data lands ~12 months later, resetting the clock - Worst case during steady state: age = ~30 months (just before next year drops AND publication lag at upper end) - 24mo budget catches catastrophic stalls (>2y silent upstream) without false-positive paging during normal between-publications ## Shape contract — third distinct shape this sprint Per-country dict where each country has its OWN year (different from Sprint 2/3a per-item arrays AND from Sprint 3b single-snapshot period): {countries: {US: {value, year: 2024}, KW: {year: 2021}, ...}, seededAt} `newestItemAt` = end-of-(max year across all countries) — drives staleness. Late reporters (KW/QA/AE) lagging G7 don't drag the panel into STALE_CONTENT; once any country's year advances, the clock resets. `oldestItemAt` = end-of-(min year across countries) — informational. ## Pattern parity with Sprint 2/3a/3b Pure helpers in `scripts/_power-reliability-helpers.mjs`: `yearToEndOfYearMs`, `powerReliabilityContentMeta` (with injectable `nowMs`), `POWER_RELIABILITY_MAX_CONTENT_AGE_MIN`. Seeder imports; test imports. No replicas. `seed-power-reliability.mjs` is NOT in Dockerfile.relay (verified via grep), so no COPY-line update needed. ## Verification - 14/14 power-reliability content-age tests pass - 46/46 across Sprint 1 + Sprint 4 stack - typecheck:api clean; lint clean - Tests include a dedicated `fresh-arrival regression guard` test that pins the EXACT budget/natural-lag mismatch failure mode (Sprint 3b lesson made concrete) so a future budget tightening cannot silently re-introduce the immediate-page bug - Boundary test: 2023 data in May 2026 (~29mo) DOES trip — confirms the staleness clock works correctly past the budget threshold * fix(power-reliability): bump budget 24mo→36mo to cover steady-state ceiling Greptile P1 on PR koala73#3602: 24-month budget false-positives mid-cycle when next-year data publishes legitimately late. The math I missed in the initial commit: fresh-arrival lag (~17mo for WB EG.ELC.LOSS.ZS) is the FLOOR but not the worst case. Once year N is in cache, it stays there until year N+1 publishes — which can legitimately take up to end-of-(N+1) + 18mo = end-of-N + 30mo under the documented 12-18 month publication-lag range. So cache age can reach 30 months between publications WITHOUT any real upstream stall. Corrected budget = 30mo steady-state ceiling + 6mo slack = 36 months (36 thirty-day-months = 1080 days ≈ 3 years). Also resolves the P2 prose-vs-math mismatch (JSDoc previously said "730 days" but `24 * 30 * 24 * 60` = 720; new wording "36 thirty-day months ≈ 1080 days" is internally consistent). General formula now documented in the helper JSDoc: budget >= max_publication_lag + cycle_length + slack Both halves required: fresh-arrival lag AND cycle_length. Initial PR covered fresh-arrival (~17mo) but missed cycle_length (12mo), which is exactly how the false-positive emerges. Same shape as Sprint 3b PR koala73#3599 P1 — that one missed fresh-arrival; this one missed steady-state. Tests: - Renamed boundary test "max year 2023 (~29mo) DOES trip" → "steady-state regression guard: max year 2023 (~29mo) does NOT trip — within ceiling" with assertion direction flipped (29mo < 30mo ceiling = legitimate late-publication wait, not staleness) - Added new boundary test "max year 2022 (~40mo) DOES trip — past ceiling = real stall" to confirm the budget fires correctly past the ceiling - Constant assertion: 36 * 30 * 24 * 60 15/15 power-reliability tests pass; 47/47 across Sprint 1+4 stack; typecheck:api clean; lint clean.
…ssil-share (koala73#3603) * feat(wb-cohort): Sprint 4 follow-up — content-age for low-carbon + fossil-share Sprint 4 cohort follow-up of the 2026-05-04 health-readiness probe plan. Migrates the two remaining WB resilience seeders that match power-reliability's shape: seed-low-carbon-generation.mjs and seed-fossil-electricity-share.mjs. Branched off Sprint 1 (koala73#3596) as a parallel sibling. ## Why a shared helper this time Three production seeders now use the IDENTICAL per-country-dict shape ({countries: {ISO2: {value, year}}, seededAt}) with the IDENTICAL contentMeta math (max-year selection + end-of-year UTC + 1h skew limit). Per CLAUDE.md "three similar lines is better than a premature abstraction" — three is exactly the line for justifying the abstraction now. New `scripts/_wb-country-dict-content-age-helpers.mjs` exports: - yearToEndOfYearMs(year) - wbCountryDictContentMeta(data, nowMs?) Each seeder imports it + brings its own MAX_CONTENT_AGE_MIN constant inline (per-seeder budgets matter — see below). seed-power-reliability keeps its own helper for now (PR koala73#3602 is in review; backporting to the shared helper is a follow-up after merge to keep that PR's diff focused). The math is verifiably identical. ## Per-seeder budgets (NOT one-size-fits-all) Verified against live WB API on 2026-05-05 — publication lags differ across these "annual WB indicators": - low-carbon-generation (NUCL+RNEW+HYRO sum, MAX year of 3): max year = 2024 (driven by NUCL/HYRO; RNEW lags to 2021 but is masked by MAX-of-3 in the seeder's countries[iso2].year compute) → fresh-arrival lag ~17mo → 36mo budget (= 30mo steady-state ceiling + 6mo slack) → matches power-reliability exactly - fossil-electricity-share (EG.ELC.FOSL.ZS): max year = 2023 (NOT 2024 — slower-publishing indicator) → fresh-arrival lag ~29mo → 48mo budget (= 41mo steady-state ceiling + 7mo slack) A naive cohort-wide budget would either false-positive on fossil-share (if 36mo) or be wastefully loose on low-carbon (if 48mo). Per-seeder constants are the correct response — each indicator's lag is empirically different. The "per-seeder budget separation" test pins this explicitly: a 41mo cache trips low-carbon (36mo) but NOT fossil-share (48mo). Demonstrates that the budgets aren't accidental — they reflect real upstream cadence differences. ## Renewables (RNEW.ZS) data-quality flag Discovered during the audit: EG.ELC.RNEW.ZS max year = 2021 in May 2026, ~53mo lag. Inside low-carbon-generation it's masked by MAX(NUCL, RNEW, HYRO), so content-age looks fine. But the underlying renewable share data is genuinely 5+ years stale. Not addressed in this PR — flagging as a separate data-quality concern for follow-up review. ## Verification - 15/15 wb-country-dict content-age tests pass (incl. fresh-arrival + steady-state regression guards for BOTH new seeders, plus a per-seeder budget separation test) - 47/47 across Sprint 1 + cohort follow-up stack - typecheck:api clean; lint clean - Neither seeder is in Dockerfile.relay (verified via grep) — no relay-COPY change needed Sprint 4 is now done for the WB cohort (3 of 5 plan-listed indicators migrated, with a 4th — IMF/WEO — explicitly deferred because it has forecast-year semantics that need different content-age handling). * fix: address Greptile PR koala73#3603 P2 nits (misleading comment + import order) P2 — `tests/wb-country-dict-content-age.test.mjs:79` — misleading inline comment: read `// end-of-2026 = Dec 31 23:59:59 = past FIXED_NOW (May 5)` but FIXED_NOW is 2026-05-05 and end-of-2026 is ~7 months in the FUTURE, not past. The test logic is correct (the EDGE year IS excluded as future-dated beyond skew tolerance) — only the comment was wrong. P2 — `scripts/seed-fossil-electricity-share.mjs:30` — `import iso3ToIso2` appeared on the line immediately after `const MAX_CONTENT_AGE_MIN`. ES module `import`s are hoisted regardless of source order, but interleaving with declarations confuses readers (code "looks" sequential but the import actually executes first). Moved the import up alongside the other top-of-module imports. Both pure-text nits — no behavior change. typecheck clean; targeted tests/wb-country-dict-content-age.test.mjs passes 15/15.
…tics, 18mo budget) (koala73#3604) * feat(imf-weo): Sprint 4 IMF cohort — content-age (forecast-year semantics, 18mo budget) Closes the deferred IMF/WEO portion of Sprint 4 (plan §477-485 listed "plus IMF/WEO/etc." as part of the annual-data migration). Branched off Sprint 1 (koala73#3596) as a parallel sibling. Migrates all 4 IMF SDMX seeders in one PR: - seed-imf-external.mjs (BCA, TM_RPCH, TX_RPCH) - seed-imf-growth.mjs (NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP) - seed-imf-labor.mjs (LUR, LP) - seed-imf-macro.mjs (PCPIPCH, BCA_NGDPD, GGR_NGDP, PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP) ## The semantic difference from WB cohort (and why a separate helper) WB indicators store the OBSERVED year — `record.date = "2024"` means data observed during calendar year 2024. The WB helper maps year → end-of-year UTC ms (the latest observation date inside the named year). IMF/WEO stores the FORECAST horizon, NOT an observation year. The `weoYears()` function in `_seed-utils.mjs` returns `[currentYear, currentYear-1, currentYear-2]` and `latestValue()` picks the first year that has a finite value. So in May 2026 after the April 2026 WEO release, max stored year = 2026 — that's IMF's freshest *forecast* for fiscal 2026, not observations through end-of-2026. If the IMF helper reused the WB cohort helper (`yearToEndOfYearMs`): year=2026 → end-of-2026 = Dec 31 2026 = ~7 months FUTURE relative to NOW → rejected by 1h skew limit → `contentMeta` returns null → every fresh IMF cache reports STALE_CONTENT. That's the failure mode this module avoids. Mapping rationale: `imfForecastYearToMs(year)` returns `Date.UTC(year - 1, 11, 31, 23, 59, 59, 999)`. Reads as: "the latest fully-observed period this forecast vintage is built on." For year=2026 → end-of-2025 = ~5 months ago in May 2026. Correctly fresh. A dedicated test (`semantic difference from WB cohort: forecast year 2026 in May 2026 maps to past (NOT future)`) exists specifically to prevent a future refactor from collapsing the WB and IMF helpers. ## Why one shared budget across all 4 IMF seeders (NOT per-seeder) WB cohort had per-seeder budgets because publication lags differed (LOSS at ~17mo, FOSL at ~29mo). All 4 IMF seeders use the IDENTICAL upstream — IMF SDMX/WEO. WEO publishes April + October vintages each year as a single integrated release covering all WorldMonitor's indicator codes. So all 4 share the same fresh-arrival lag and the same steady-state ceiling. One budget = correct. ## 18-month budget — derivation Steady-state model under "year → end-of-(year-1)" mapping: - After April N release: max year = N → newestItemAt = end-of-(N-1). Age = ~5 months. - After October N: max year still = N → age = ~11 months. - Just before April N+1: max year still = N → age = ~16 months. - After April N+1: max year advances to N+1 → newestItemAt resets. Steady-state ceiling = 16mo (just before April release of next year). Budget = 16mo + 2mo slack = 18 months. Trips when a full year of WEO releases is missed (both April AND October vintages of one year), which is the right pager threshold for an IMF outage. ## Verification - 15/15 imf-weo content-age tests pass (incl. fresh-arrival + steady- state regression guards, future-skew defense, late-reporter cohort handling, and the WB-vs-IMF semantic-difference guard test) - Tested with `npx tsx --test` against the existing IMF test suites: 34/34 across `imf-country-data` + `seed-imf-extended` + new file - 47/47 across Sprint 1 + IMF cohort stack - typecheck:api clean; lint clean - Zero seed-imf-*.mjs files in Dockerfile.relay (verified via grep) so no relay-COPY change needed ## Sprint 4 status after this PR - ✅ power-reliability (koala73#3602) - ✅ low-carbon-generation + fossil-electricity-share (koala73#3603) - ✅ IMF/WEO cohort: external + growth + labor + macro (this PR) Plan §477-485 fully closed. The plan's "Definition of done" §530 (≥1 annual-data migrated) was satisfied by koala73#3602; this PR + koala73#3603 round out the rest of the listed cohort. * fix(imf-weo): use max forecast year for content-age, not priority-first metric Codex PR koala73#3604 P2. The four IMF/WEO seeders write `entry.year` as the priority-first non-null indicator's year (`ca?.year ?? tm?.year ?? tx?.year` in seed-imf-external). That's correct as the public payload's "primary metric vintage" but WRONG for content-age: a row with BCA=2024 + import-volume=2026 publishes year=2024, even though the country dict carries a fresh 2026 metric — content-age maps it to 2023-12-31 (~17mo old, near-stale) when it actually carries a 2026 metric (~5mo old in May 2026). Fix path A (preserves public payload semantics): seeders now populate a dedicated `latestYear` field via a new `maxIntegerYear()` helper, computed across ALL the country's indicator years. The content-age helper prefers `entry.latestYear` over `entry.year`, falling back to `year` for back-compat with caches written before this PR. - scripts/_imf-weo-content-age-helpers.mjs — export `maxIntegerYear()`; `imfWeoContentMeta` reads `entry.latestYear` first - scripts/seed-imf-{external,growth,labor,macro}.mjs — populate `latestYear` alongside existing `year` (no public payload change beyond the new field) - tests/imf-weo-content-age.test.mjs — add maxIntegerYear unit tests + three mixed-indicator-year regression tests covering the fresh-metric- behind-stale-primary case, latestYear=null fallback, and heterogeneous cohort newest/oldest extraction * chore(imf-weo): adversarial-review hardening — horizon-extension trap guard + schemaVersion bump PR koala73#3604 review findings #1 + #2. Both advisory, no behavior change today. #1 Horizon-extension trap: weoYears() currently returns [currentYear, currentYear-1, currentYear-2], so max year = currentYear and the 1h skew filter is purely defensive. If a future Sprint extends weoYears() to include currentYear+1 to surface forward forecasts, the skew filter would silently drop every fresh +1 entry, regressing cohort newestItemAt to the prior year and producing FALSE STALE_CONTENT for genuinely-fresh data. Added load-bearing comment near the skew check plus a regression-guard test that documents the trap shape under FIXED_NOW=2026-05-05. Test asserts the trap, not desired behavior; when horizon extension lands the test fails and forces revisit. #2 schemaVersion bump 1->2 across all 4 seeders. Codex P2 added the latestYear field; envelope newestItemAt math now differs under the same schema number. Bumping forces a clean republish on rollout and makes rollback observable rather than silently drifting envelope math while caches keep the new shape.
Sprint 4 of the 2026-05-04 health-readiness probe plan. Closes the plan's "Definition of done" line 530: at least 1 annual-data seeder migrated. Branched off Sprint 1 (#3596) as a parallel sibling to Sprints 2/3a/3b.
Summary
Why 24 months (NOT the plan's 13)
Plan §477-485 originally proposed 13 months but that's structurally wrong for WB indicators. Verified against live WB API on 2026-05-05:
```
curl https://api.worldbank.org/v2/country/USA;CHN;...;KWT/indicator/EG.ELC.LOSS.ZS
```
On that date G7 max year = 2024. End-of-2024 = Dec 31 2024 = ~17 months before fresh-arrival. WB year-N data lands in cache 12-18 months after end-of-N. A 13-month budget would have tripped `STALE_CONTENT` immediately on every successful fresh seed run — same failure mode Greptile P1 caught on Sprint 3b PR #3599 (45d budget vs 60d natural M+2 lag).
24-month math:
Shape contract — third distinct shape this sprint
Per-country dict where each country has its OWN year (different from Sprint 2/3a per-item arrays AND from Sprint 3b single-snapshot period):
```js
{ countries: { US: { value, year: 2024 }, KW: { year: 2021 }, ... }, seededAt }
```
`newestItemAt` = end-of-(max year across all countries). Late-reporters (KW/QA/AE) lagging G7 do NOT drag the panel into STALE_CONTENT — once any country's year advances, the clock resets. `oldestItemAt` = end-of-(min year), informational.
Defensive (each tested):
Test plan
Sprint 4 cohort note
This PR migrates 1 of the 4 indicators the plan listed (power-losses, low-carbon-generation, fossil-electricity-share, IMF/WEO). The other 3 are NOT in this PR — each needs its own date-source audit (some WB indicators publish quarterly with much shorter lag; the 24-month budget chosen here for EG.ELC.LOSS.ZS isn't blindly transferable). Per plan rollout §493-498: "PR 4 land last; close out the canonical-envelope-mirror story." Plan's "Definition of done" requires `>= 1` annual-data migration, which this PR satisfies; further annual indicators are follow-up work.