Skip to content

feat(iea-oil-stocks): Sprint 3b — content-age probe (45d budget)#3599

Merged
koala73 merged 2 commits into
mainfrom
feat/health-readiness-content-age-sprint-3b-iea-oil-stocks
May 8, 2026
Merged

feat(iea-oil-stocks): Sprint 3b — content-age probe (45d budget)#3599
koala73 merged 2 commits into
mainfrom
feat/health-readiness-content-age-sprint-3b-iea-oil-stocks

Conversation

@koala73

@koala73 koala73 commented May 5, 2026

Copy link
Copy Markdown
Owner

Sprint 3b of the 2026-05-04 health-readiness probe plan. Branched off Sprint 1 (#3596) as a parallel sibling to Sprint 2 (#3597) and Sprint 3a (#3598) per the plan's "Each PR is independently shippable" note.

Summary

  • Adds a content-age contract on seed-iea-oil-stocks.mjs so /api/health surfaces STALE_CONTENT when data.dataMonth is more than 45 days stale (i.e. the IEA monthly publication cycle has stalled).
  • Pure helper in scripts/_iea-oil-stocks-helpers.mjs: dataMonthToEndOfMonthMs, ieaOilStocksContentMeta (with injectable nowMs), IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN. Seeder imports it; tests import it — no replicas (matches Sprint 2/3a post-refactor pattern).
  • No Dockerfile.relay change needed — seed-iea-oil-stocks.mjs is not COPY'd into the relay container (verified via grep; relay closure test passes unchanged).

Why 45 days

IEA monthly oil stocks reports publish on an M+2 cadence — August data ships in late October/early November. The dataMonth in cache is therefore typically ~30-60 days old at "fresh-arrival" time. A 45-day budget tolerates one normal publication delay (~30d slack on top of the natural M+2 lag) and trips when a publication month is missed entirely (e.g. cache stuck at `"2024-08"` past Dec 1 when `"2024-10"` should have landed).

Shape contract — different from Sprint 2/3a

IEA is a single-snapshot seeder: every member shares one top-level dataMonth string ("YYYY-MM"), there is no per-item published-at. The new helper parses dataMonth → end-of-month UTC ms (the latest possible observation date in the named period) and returns it as both newestItemAt and oldestItemAt for Sprint 1's mirror-parity contract.

Defensive behaviors (each tested):

  • Missing/malformed dataMonth (`"2024-13"`, `"2024-8"` single-digit, non-string, etc.) → `null`
  • Future-dated dataMonth beyond 1h clock-skew tolerance → `null` (guards against upstream `yearMonth` garbage producing e.g. a `2099-12`)
  • Leap-year correctness: `"2024-02"` → Feb 29; `"2023-02"` → Feb 28
  • Month rollover safety: `"2024-12"` → Dec 31

Test plan

  • `node --test tests/iea-oil-stocks-content-age.test.mjs` → 15/15 pass
  • `node --test tests/iea-oil-stocks-seed.test.mjs tests/iea-oil-stocks-content-age.test.mjs tests/seed-content-age-contract.test.mjs tests/health-content-age.test.mjs tests/seed-utils-empty-data-failure.test.mjs` → 78/78 pass
  • `node --test tests/dockerfile-relay-imports.test.mjs` → 3/3 pass (no relay impact)
  • `npm run typecheck:api` clean
  • `npm run lint` clean (pre-existing warnings only)
  • Post-merge: confirm /api/health snapshot shows `oilStocks` with `contentAge` block populated, `status: OK`, and contentAgeMin matching Date.now() - end-of-dataMonth
  • Post-merge: simulate stale (manually rewrite the canonical key with an old dataMonth in a preview env) → confirm STALE_CONTENT

@vercel

vercel Bot commented May 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment May 8, 2026 9:02am

Request Review

@greptile-apps

greptile-apps Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds a content-age probe for the IEA oil stocks seeder by introducing a new pure-helper module (_iea-oil-stocks-helpers.mjs) that parses the top-level dataMonth: "YYYY-MM" string into end-of-month UTC milliseconds and wires it into runSeed via contentMeta/maxContentAgeMin. The overall design is clean and follows the Sprint 2/3a pattern.

  • New helper module exports dataMonthToEndOfMonthMs (end-of-month UTC ms with full input validation), ieaOilStocksContentMeta (skew-guarded, injectable-nowMs wrapper), and IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN (45 days); the single-snapshot newestItemAt === oldestItemAt shape matches the Sprint 1 contract.
  • Seeder change is a two-line addition passing the helpers into runSeed; no other seeder logic is touched and the relay closure test is unaffected.
  • Test suite covers parsing validity, leap-year correctness, skew-limit rejection, and pilot-threshold sanity across 15 deterministic cases, though some test titles misstate the actual data age relative to FIXED_NOW.

Confidence Score: 3/5

Safe to merge if the team accepts that STALE_CONTENT may fire during the second half of the normal inter-publication window; needs explicit confirmation if the intent is for the check to remain silent during a fully normal IEA publication cycle.

The seeder wiring and helper logic are correct — parsing, leap-year handling, and skew-limit guards all work as shown by the tests. The concern is that the helper's own JSDoc acknowledges fresh data is typically 30-60 days old at fresh-arrival time, yet the threshold is set at 45 days. For any fresh delivery in the 46-60 day range, the health endpoint would report STALE_CONTENT immediately after a successful seed run. The test fixture for September data at FIXED_NOW = Nov 14 sits at ~44.9 days — a single week's publication delay would push fresh data over the limit and make the alert fire persistently through the entire next inter-publication period.

scripts/_iea-oil-stocks-helpers.mjs — the 45-day constant and its JSDoc warrant a second look against actual IEA arrival-age data from production.

Important Files Changed

Filename Overview
scripts/_iea-oil-stocks-helpers.mjs New helper module exporting dataMonthToEndOfMonthMs, ieaOilStocksContentMeta, and IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN; the 45-day threshold may be too tight relative to the stated M+2 publication cadence where fresh data can arrive 55+ days old.
scripts/seed-iea-oil-stocks.mjs Adds contentMeta and maxContentAgeMin to the runSeed call by importing from the new helper module; change is minimal and structurally consistent with Sprint 2/3a pattern.
tests/iea-oil-stocks-content-age.test.mjs 15 deterministic tests covering parsing, leap-year, skew-limit, and threshold sanity; two test titles misstate the actual data age (60d vs ~75d actual; ~30d vs ~14d actual) and one comment misidentifies the M+2 publication month for September data.

Sequence Diagram

sequenceDiagram
    participant Seeder as seed-iea-oil-stocks.mjs
    participant Helper as _iea-oil-stocks-helpers.mjs
    participant RunSeed as runSeed()
    participant Health as /api/health

    Seeder->>Helper: import ieaOilStocksContentMeta, IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN
    Seeder->>RunSeed: runSeed(..., { contentMeta, maxContentAgeMin: 64800 })
    RunSeed->>Helper: ieaOilStocksContentMeta(data, Date.now())
    Helper->>Helper: dataMonthToEndOfMonthMs(data.dataMonth)
    note over Helper: YYYY-MM → end-of-month UTC ms<br/>null on invalid/future input
    Helper-->>RunSeed: { newestItemAt: ts, oldestItemAt: ts } | null
    RunSeed->>Health: writes seed-meta with contentAge block
    Health->>Health: contentAgeMin = (now - newestItemAt) / 60000
    alt contentAgeMin > 64800 (45 days)
        Health-->>Health: status: STALE_CONTENT
    else within budget
        Health-->>Health: status: OK
    end
Loading

Reviews (1): Last reviewed commit: "feat(iea-oil-stocks): Sprint 3b — conten..." | Re-trigger Greptile

Comment thread scripts/_iea-oil-stocks-helpers.mjs Outdated
Comment thread tests/iea-oil-stocks-content-age.test.mjs Outdated
Comment thread tests/iea-oil-stocks-content-age.test.mjs Outdated
Comment thread tests/iea-oil-stocks-content-age.test.mjs Outdated
koala73 added a commit that referenced this pull request May 5, 2026
Greptile P1 on PR #3599: a 45-day budget contradicts the helper's own
M+2 cadence claim. End-of-observation-month (Aug 31) is ~60-65 days
BEFORE publication (~late Oct/early Nov), so fresh-arrival data is
already past the 45d threshold at the moment a successful seed run
writes it. STALE_CONTENT would have fired on every cron tick.

Corrected math: 90d = ~60d natural M+2 lag + ~30d missed-publication
slack. Trips only when a month is missed entirely (cache stuck at
"2024-08" past mid-Jan when "2024-10" should have landed).

Also addresses 3 P2 review nits in the same edit:

- Test "60 days old" → "fresh-arrival regression guard: ~60d-old fresh
  M+2 data does NOT trip" (the math was right, name was wrong; rewrote
  the test to actually pin the failure mode the P1 cited).
- Test "~30 days old" → "~14 days old" (the fixture was "2023-10" =
  ~14d before FIXED_NOW, not 30).
- M+2 lag scenario comment "Sept data published ~Oct 25" → "~late Nov
  (M+2 cadence)" — Oct 25 is M+1, not M+2.

Added: dedicated fresh-arrival regression guard test that asserts a
~75d-old fresh M+2 dataMonth is within budget. Without it, a future
budget tightening could re-introduce the immediate-page bug invisibly.

Verification: 16/16 iea content-age (was 15/15 — added regression guard);
79/79 across iea seed + Sprint 1 + Sprint 3b stack; typecheck:api clean.
koala73 added a commit that referenced this pull request May 5, 2026
…eiling

Greptile P1 on PR #3602: 24-month budget false-positives mid-cycle when
next-year data publishes legitimately late.

The math I missed in the initial commit: fresh-arrival lag (~17mo for WB
EG.ELC.LOSS.ZS) is the FLOOR but not the worst case. Once year N is in
cache, it stays there until year N+1 publishes — which can legitimately
take up to end-of-(N+1) + 18mo = end-of-N + 30mo under the documented
12-18 month publication-lag range. So cache age can reach 30 months
between publications WITHOUT any real upstream stall.

Corrected budget = 30mo steady-state ceiling + 6mo slack = 36 months
(36 thirty-day-months = 1080 days ≈ 3 years).

Also resolves the P2 prose-vs-math mismatch (JSDoc previously said
"730 days" but `24 * 30 * 24 * 60` = 720; new wording "36 thirty-day
months ≈ 1080 days" is internally consistent).

General formula now documented in the helper JSDoc:

  budget >= max_publication_lag + cycle_length + slack

Both halves required: fresh-arrival lag AND cycle_length. Initial PR
covered fresh-arrival (~17mo) but missed cycle_length (12mo), which is
exactly how the false-positive emerges. Same shape as Sprint 3b PR #3599
P1 — that one missed fresh-arrival; this one missed steady-state.

Tests:
- Renamed boundary test "max year 2023 (~29mo) DOES trip" → "steady-state
  regression guard: max year 2023 (~29mo) does NOT trip — within ceiling"
  with assertion direction flipped (29mo < 30mo ceiling = legitimate
  late-publication wait, not staleness)
- Added new boundary test "max year 2022 (~40mo) DOES trip — past ceiling
  = real stall" to confirm the budget fires correctly past the ceiling
- Constant assertion: 36 * 30 * 24 * 60

15/15 power-reliability tests pass; 47/47 across Sprint 1+4 stack;
typecheck:api clean; lint clean.
Base automatically changed from feat/health-readiness-content-age-sprint-1 to main May 8, 2026 08:43
koala73 added 2 commits May 8, 2026 12:55
Sparse seeders sub-PR b/c of the 2026-05-04 health-readiness plan.
Branched off Sprint 1 (#3596) as a parallel sibling to Sprint 2 (#3597)
and Sprint 3a (#3598) per the plan's "Each PR is independently shippable"
note (line 498).

## Why this matters

IEA monthly oil stocks publish on an M+2 cadence — August data ships in
late October/early November. Without a content-age probe, a stalled
publication month is invisible to /api/health: the seeder runs fine on
its 6h cron, fetchedAt stays fresh, but data.dataMonth never advances. A
45-day budget trips STALE_CONTENT exactly when a month has been missed
(e.g. cache shows "2024-08" past Dec 1 when "2024-10" should have
landed).

## Shape contract — different from Sprint 2/3a

IEA is a SINGLE-SNAPSHOT seeder: every member shares one `dataMonth`
("YYYY-MM" string at the top level), there is no per-item published-at.
The new helper parses dataMonth → end-of-month UTC ms (the latest
possible observation date in the named period) and returns it as both
`newestItemAt` and `oldestItemAt`.

Defensive: contentMeta returns null when dataMonth is missing, malformed
("2024-13", "2024-8" single-digit), or future-dated beyond 1h clock-skew
tolerance (guards against upstream yearMonth garbage producing e.g. a
2099-12 dataMonth).

## Pattern parity with Sprint 2/3a

Following the established pattern: pure helpers in
`scripts/_iea-oil-stocks-helpers.mjs` (`dataMonthToEndOfMonthMs`,
`ieaOilStocksContentMeta`, `IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN`).
Seeder imports them; tests import them. No replicas.

`seed-iea-oil-stocks.mjs` is NOT in Dockerfile.relay (verified via
`grep`), so no COPY-line update needed (unlike Sprint 3a's
seed-climate-news which IS relay-COPY'd).

## Verification

- 15/15 iea content-age tests pass (incl. leap-year, month-rollover,
  invalid-shape rejection, M+2 lag realism, future-clock-skew defense)
- 78/78 across iea seed + Sprint 1 + Sprint 3b stack
- typecheck:api clean; lint clean (pre-existing warnings only)
- Dockerfile.relay closure test passes (no relay impact)
Greptile P1 on PR #3599: a 45-day budget contradicts the helper's own
M+2 cadence claim. End-of-observation-month (Aug 31) is ~60-65 days
BEFORE publication (~late Oct/early Nov), so fresh-arrival data is
already past the 45d threshold at the moment a successful seed run
writes it. STALE_CONTENT would have fired on every cron tick.

Corrected math: 90d = ~60d natural M+2 lag + ~30d missed-publication
slack. Trips only when a month is missed entirely (cache stuck at
"2024-08" past mid-Jan when "2024-10" should have landed).

Also addresses 3 P2 review nits in the same edit:

- Test "60 days old" → "fresh-arrival regression guard: ~60d-old fresh
  M+2 data does NOT trip" (the math was right, name was wrong; rewrote
  the test to actually pin the failure mode the P1 cited).
- Test "~30 days old" → "~14 days old" (the fixture was "2023-10" =
  ~14d before FIXED_NOW, not 30).
- M+2 lag scenario comment "Sept data published ~Oct 25" → "~late Nov
  (M+2 cadence)" — Oct 25 is M+1, not M+2.

Added: dedicated fresh-arrival regression guard test that asserts a
~75d-old fresh M+2 dataMonth is within budget. Without it, a future
budget tightening could re-introduce the immediate-page bug invisibly.

Verification: 16/16 iea content-age (was 15/15 — added regression guard);
79/79 across iea seed + Sprint 1 + Sprint 3b stack; typecheck:api clean.
koala73 added a commit that referenced this pull request May 8, 2026
Closes the plan's "Definition of done" item: at least 1 annual-data
seeder migrated. Branched off Sprint 1 (#3596) as a parallel sibling
to Sprints 2/3a/3b.

## Why this matters

WB EG.ELC.LOSS.ZS publishes annually. Without a content-age probe, a
stalled WB publication cycle is invisible to /api/health: the seeder
runs fine on its 35-day TTL, fetchedAt stays fresh, but no country's
year ever advances past e.g. 2024. STALE_CONTENT trips correctly when
the cache stops advancing — for power-reliability, that means "by the
time you'd expect year-N+1 data, year-N is still latest" → page on-call.

## Why 24 months (NOT the plan's 13 months)

Plan §477-485 originally proposed `13 * 30 * 24 * 60` minutes (~13
months), but this is structurally wrong for WB indicators — verified
against live WB API on 2026-05-05:

  curl https://api.worldbank.org/v2/country/USA;CHN;...;KWT/indicator/EG.ELC.LOSS.ZS

On that date G7 max year = 2024. End-of-2024 = Dec 31 2024 = ~17 months
before the seed. WB year-N data lands in cache 12-18 months after
end-of-N (publication lag varies). A 13-month budget would have tripped
STALE_CONTENT immediately on every successful fresh-arrival — the same
failure mode Greptile P1 caught on Sprint 3b PR #3599 (45d budget vs
M+2 60-day natural lag).

24mo math:
  - Year N data lands at age = 12-18 months (publication lag)
  - Year (N+1) data lands ~12 months later, resetting the clock
  - Worst case during steady state: age = ~30 months (just before next
    year drops AND publication lag at upper end)
  - 24mo budget catches catastrophic stalls (>2y silent upstream)
    without false-positive paging during normal between-publications

## Shape contract — third distinct shape this sprint

Per-country dict where each country has its OWN year (different from
Sprint 2/3a per-item arrays AND from Sprint 3b single-snapshot period):

  {countries: {US: {value, year: 2024}, KW: {year: 2021}, ...}, seededAt}

`newestItemAt` = end-of-(max year across all countries) — drives
staleness. Late reporters (KW/QA/AE) lagging G7 don't drag the panel
into STALE_CONTENT; once any country's year advances, the clock resets.

`oldestItemAt` = end-of-(min year across countries) — informational.

## Pattern parity with Sprint 2/3a/3b

Pure helpers in `scripts/_power-reliability-helpers.mjs`:
`yearToEndOfYearMs`, `powerReliabilityContentMeta` (with injectable
`nowMs`), `POWER_RELIABILITY_MAX_CONTENT_AGE_MIN`. Seeder imports;
test imports. No replicas.

`seed-power-reliability.mjs` is NOT in Dockerfile.relay (verified via
grep), so no COPY-line update needed.

## Verification

- 14/14 power-reliability content-age tests pass
- 46/46 across Sprint 1 + Sprint 4 stack
- typecheck:api clean; lint clean
- Tests include a dedicated `fresh-arrival regression guard` test
  that pins the EXACT budget/natural-lag mismatch failure mode
  (Sprint 3b lesson made concrete) so a future budget tightening
  cannot silently re-introduce the immediate-page bug
- Boundary test: 2023 data in May 2026 (~29mo) DOES trip — confirms
  the staleness clock works correctly past the budget threshold
koala73 added a commit that referenced this pull request May 8, 2026
…eiling

Greptile P1 on PR #3602: 24-month budget false-positives mid-cycle when
next-year data publishes legitimately late.

The math I missed in the initial commit: fresh-arrival lag (~17mo for WB
EG.ELC.LOSS.ZS) is the FLOOR but not the worst case. Once year N is in
cache, it stays there until year N+1 publishes — which can legitimately
take up to end-of-(N+1) + 18mo = end-of-N + 30mo under the documented
12-18 month publication-lag range. So cache age can reach 30 months
between publications WITHOUT any real upstream stall.

Corrected budget = 30mo steady-state ceiling + 6mo slack = 36 months
(36 thirty-day-months = 1080 days ≈ 3 years).

Also resolves the P2 prose-vs-math mismatch (JSDoc previously said
"730 days" but `24 * 30 * 24 * 60` = 720; new wording "36 thirty-day
months ≈ 1080 days" is internally consistent).

General formula now documented in the helper JSDoc:

  budget >= max_publication_lag + cycle_length + slack

Both halves required: fresh-arrival lag AND cycle_length. Initial PR
covered fresh-arrival (~17mo) but missed cycle_length (12mo), which is
exactly how the false-positive emerges. Same shape as Sprint 3b PR #3599
P1 — that one missed fresh-arrival; this one missed steady-state.

Tests:
- Renamed boundary test "max year 2023 (~29mo) DOES trip" → "steady-state
  regression guard: max year 2023 (~29mo) does NOT trip — within ceiling"
  with assertion direction flipped (29mo < 30mo ceiling = legitimate
  late-publication wait, not staleness)
- Added new boundary test "max year 2022 (~40mo) DOES trip — past ceiling
  = real stall" to confirm the budget fires correctly past the ceiling
- Constant assertion: 36 * 30 * 24 * 60

15/15 power-reliability tests pass; 47/47 across Sprint 1+4 stack;
typecheck:api clean; lint clean.
@koala73
koala73 force-pushed the feat/health-readiness-content-age-sprint-3b-iea-oil-stocks branch from a3f89a4 to f90a484 Compare May 8, 2026 08:58
@koala73
koala73 merged commit be5061e into main May 8, 2026
11 checks passed
@koala73
koala73 deleted the feat/health-readiness-content-age-sprint-3b-iea-oil-stocks branch May 8, 2026 09:06
koala73 added a commit that referenced this pull request May 8, 2026
…t) (#3602)

* feat(power-reliability): Sprint 4 — content-age probe (24-month budget)

Closes the plan's "Definition of done" item: at least 1 annual-data
seeder migrated. Branched off Sprint 1 (#3596) as a parallel sibling
to Sprints 2/3a/3b.

## Why this matters

WB EG.ELC.LOSS.ZS publishes annually. Without a content-age probe, a
stalled WB publication cycle is invisible to /api/health: the seeder
runs fine on its 35-day TTL, fetchedAt stays fresh, but no country's
year ever advances past e.g. 2024. STALE_CONTENT trips correctly when
the cache stops advancing — for power-reliability, that means "by the
time you'd expect year-N+1 data, year-N is still latest" → page on-call.

## Why 24 months (NOT the plan's 13 months)

Plan §477-485 originally proposed `13 * 30 * 24 * 60` minutes (~13
months), but this is structurally wrong for WB indicators — verified
against live WB API on 2026-05-05:

  curl https://api.worldbank.org/v2/country/USA;CHN;...;KWT/indicator/EG.ELC.LOSS.ZS

On that date G7 max year = 2024. End-of-2024 = Dec 31 2024 = ~17 months
before the seed. WB year-N data lands in cache 12-18 months after
end-of-N (publication lag varies). A 13-month budget would have tripped
STALE_CONTENT immediately on every successful fresh-arrival — the same
failure mode Greptile P1 caught on Sprint 3b PR #3599 (45d budget vs
M+2 60-day natural lag).

24mo math:
  - Year N data lands at age = 12-18 months (publication lag)
  - Year (N+1) data lands ~12 months later, resetting the clock
  - Worst case during steady state: age = ~30 months (just before next
    year drops AND publication lag at upper end)
  - 24mo budget catches catastrophic stalls (>2y silent upstream)
    without false-positive paging during normal between-publications

## Shape contract — third distinct shape this sprint

Per-country dict where each country has its OWN year (different from
Sprint 2/3a per-item arrays AND from Sprint 3b single-snapshot period):

  {countries: {US: {value, year: 2024}, KW: {year: 2021}, ...}, seededAt}

`newestItemAt` = end-of-(max year across all countries) — drives
staleness. Late reporters (KW/QA/AE) lagging G7 don't drag the panel
into STALE_CONTENT; once any country's year advances, the clock resets.

`oldestItemAt` = end-of-(min year across countries) — informational.

## Pattern parity with Sprint 2/3a/3b

Pure helpers in `scripts/_power-reliability-helpers.mjs`:
`yearToEndOfYearMs`, `powerReliabilityContentMeta` (with injectable
`nowMs`), `POWER_RELIABILITY_MAX_CONTENT_AGE_MIN`. Seeder imports;
test imports. No replicas.

`seed-power-reliability.mjs` is NOT in Dockerfile.relay (verified via
grep), so no COPY-line update needed.

## Verification

- 14/14 power-reliability content-age tests pass
- 46/46 across Sprint 1 + Sprint 4 stack
- typecheck:api clean; lint clean
- Tests include a dedicated `fresh-arrival regression guard` test
  that pins the EXACT budget/natural-lag mismatch failure mode
  (Sprint 3b lesson made concrete) so a future budget tightening
  cannot silently re-introduce the immediate-page bug
- Boundary test: 2023 data in May 2026 (~29mo) DOES trip — confirms
  the staleness clock works correctly past the budget threshold

* fix(power-reliability): bump budget 24mo→36mo to cover steady-state ceiling

Greptile P1 on PR #3602: 24-month budget false-positives mid-cycle when
next-year data publishes legitimately late.

The math I missed in the initial commit: fresh-arrival lag (~17mo for WB
EG.ELC.LOSS.ZS) is the FLOOR but not the worst case. Once year N is in
cache, it stays there until year N+1 publishes — which can legitimately
take up to end-of-(N+1) + 18mo = end-of-N + 30mo under the documented
12-18 month publication-lag range. So cache age can reach 30 months
between publications WITHOUT any real upstream stall.

Corrected budget = 30mo steady-state ceiling + 6mo slack = 36 months
(36 thirty-day-months = 1080 days ≈ 3 years).

Also resolves the P2 prose-vs-math mismatch (JSDoc previously said
"730 days" but `24 * 30 * 24 * 60` = 720; new wording "36 thirty-day
months ≈ 1080 days" is internally consistent).

General formula now documented in the helper JSDoc:

  budget >= max_publication_lag + cycle_length + slack

Both halves required: fresh-arrival lag AND cycle_length. Initial PR
covered fresh-arrival (~17mo) but missed cycle_length (12mo), which is
exactly how the false-positive emerges. Same shape as Sprint 3b PR #3599
P1 — that one missed fresh-arrival; this one missed steady-state.

Tests:
- Renamed boundary test "max year 2023 (~29mo) DOES trip" → "steady-state
  regression guard: max year 2023 (~29mo) does NOT trip — within ceiling"
  with assertion direction flipped (29mo < 30mo ceiling = legitimate
  late-publication wait, not staleness)
- Added new boundary test "max year 2022 (~40mo) DOES trip — past ceiling
  = real stall" to confirm the budget fires correctly past the ceiling
- Constant assertion: 36 * 30 * 24 * 60

15/15 power-reliability tests pass; 47/47 across Sprint 1+4 stack;
typecheck:api clean; lint clean.
fuleinist pushed a commit to fuleinist/worldmonitor that referenced this pull request May 9, 2026
…la73#3599)

* feat(iea-oil-stocks): Sprint 3b — content-age probe (45d budget)

Sparse seeders sub-PR b/c of the 2026-05-04 health-readiness plan.
Branched off Sprint 1 (koala73#3596) as a parallel sibling to Sprint 2 (koala73#3597)
and Sprint 3a (koala73#3598) per the plan's "Each PR is independently shippable"
note (line 498).

## Why this matters

IEA monthly oil stocks publish on an M+2 cadence — August data ships in
late October/early November. Without a content-age probe, a stalled
publication month is invisible to /api/health: the seeder runs fine on
its 6h cron, fetchedAt stays fresh, but data.dataMonth never advances. A
45-day budget trips STALE_CONTENT exactly when a month has been missed
(e.g. cache shows "2024-08" past Dec 1 when "2024-10" should have
landed).

## Shape contract — different from Sprint 2/3a

IEA is a SINGLE-SNAPSHOT seeder: every member shares one `dataMonth`
("YYYY-MM" string at the top level), there is no per-item published-at.
The new helper parses dataMonth → end-of-month UTC ms (the latest
possible observation date in the named period) and returns it as both
`newestItemAt` and `oldestItemAt`.

Defensive: contentMeta returns null when dataMonth is missing, malformed
("2024-13", "2024-8" single-digit), or future-dated beyond 1h clock-skew
tolerance (guards against upstream yearMonth garbage producing e.g. a
2099-12 dataMonth).

## Pattern parity with Sprint 2/3a

Following the established pattern: pure helpers in
`scripts/_iea-oil-stocks-helpers.mjs` (`dataMonthToEndOfMonthMs`,
`ieaOilStocksContentMeta`, `IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN`).
Seeder imports them; tests import them. No replicas.

`seed-iea-oil-stocks.mjs` is NOT in Dockerfile.relay (verified via
`grep`), so no COPY-line update needed (unlike Sprint 3a's
seed-climate-news which IS relay-COPY'd).

## Verification

- 15/15 iea content-age tests pass (incl. leap-year, month-rollover,
  invalid-shape rejection, M+2 lag realism, future-clock-skew defense)
- 78/78 across iea seed + Sprint 1 + Sprint 3b stack
- typecheck:api clean; lint clean (pre-existing warnings only)
- Dockerfile.relay closure test passes (no relay impact)

* fix(iea-oil-stocks): bump budget 45d→90d to cover M+2 natural lag

Greptile P1 on PR koala73#3599: a 45-day budget contradicts the helper's own
M+2 cadence claim. End-of-observation-month (Aug 31) is ~60-65 days
BEFORE publication (~late Oct/early Nov), so fresh-arrival data is
already past the 45d threshold at the moment a successful seed run
writes it. STALE_CONTENT would have fired on every cron tick.

Corrected math: 90d = ~60d natural M+2 lag + ~30d missed-publication
slack. Trips only when a month is missed entirely (cache stuck at
"2024-08" past mid-Jan when "2024-10" should have landed).

Also addresses 3 P2 review nits in the same edit:

- Test "60 days old" → "fresh-arrival regression guard: ~60d-old fresh
  M+2 data does NOT trip" (the math was right, name was wrong; rewrote
  the test to actually pin the failure mode the P1 cited).
- Test "~30 days old" → "~14 days old" (the fixture was "2023-10" =
  ~14d before FIXED_NOW, not 30).
- M+2 lag scenario comment "Sept data published ~Oct 25" → "~late Nov
  (M+2 cadence)" — Oct 25 is M+1, not M+2.

Added: dedicated fresh-arrival regression guard test that asserts a
~75d-old fresh M+2 dataMonth is within budget. Without it, a future
budget tightening could re-introduce the immediate-page bug invisibly.

Verification: 16/16 iea content-age (was 15/15 — added regression guard);
79/79 across iea seed + Sprint 1 + Sprint 3b stack; typecheck:api clean.
fuleinist pushed a commit to fuleinist/worldmonitor that referenced this pull request May 9, 2026
…t) (koala73#3602)

* feat(power-reliability): Sprint 4 — content-age probe (24-month budget)

Closes the plan's "Definition of done" item: at least 1 annual-data
seeder migrated. Branched off Sprint 1 (koala73#3596) as a parallel sibling
to Sprints 2/3a/3b.

## Why this matters

WB EG.ELC.LOSS.ZS publishes annually. Without a content-age probe, a
stalled WB publication cycle is invisible to /api/health: the seeder
runs fine on its 35-day TTL, fetchedAt stays fresh, but no country's
year ever advances past e.g. 2024. STALE_CONTENT trips correctly when
the cache stops advancing — for power-reliability, that means "by the
time you'd expect year-N+1 data, year-N is still latest" → page on-call.

## Why 24 months (NOT the plan's 13 months)

Plan §477-485 originally proposed `13 * 30 * 24 * 60` minutes (~13
months), but this is structurally wrong for WB indicators — verified
against live WB API on 2026-05-05:

  curl https://api.worldbank.org/v2/country/USA;CHN;...;KWT/indicator/EG.ELC.LOSS.ZS

On that date G7 max year = 2024. End-of-2024 = Dec 31 2024 = ~17 months
before the seed. WB year-N data lands in cache 12-18 months after
end-of-N (publication lag varies). A 13-month budget would have tripped
STALE_CONTENT immediately on every successful fresh-arrival — the same
failure mode Greptile P1 caught on Sprint 3b PR koala73#3599 (45d budget vs
M+2 60-day natural lag).

24mo math:
  - Year N data lands at age = 12-18 months (publication lag)
  - Year (N+1) data lands ~12 months later, resetting the clock
  - Worst case during steady state: age = ~30 months (just before next
    year drops AND publication lag at upper end)
  - 24mo budget catches catastrophic stalls (>2y silent upstream)
    without false-positive paging during normal between-publications

## Shape contract — third distinct shape this sprint

Per-country dict where each country has its OWN year (different from
Sprint 2/3a per-item arrays AND from Sprint 3b single-snapshot period):

  {countries: {US: {value, year: 2024}, KW: {year: 2021}, ...}, seededAt}

`newestItemAt` = end-of-(max year across all countries) — drives
staleness. Late reporters (KW/QA/AE) lagging G7 don't drag the panel
into STALE_CONTENT; once any country's year advances, the clock resets.

`oldestItemAt` = end-of-(min year across countries) — informational.

## Pattern parity with Sprint 2/3a/3b

Pure helpers in `scripts/_power-reliability-helpers.mjs`:
`yearToEndOfYearMs`, `powerReliabilityContentMeta` (with injectable
`nowMs`), `POWER_RELIABILITY_MAX_CONTENT_AGE_MIN`. Seeder imports;
test imports. No replicas.

`seed-power-reliability.mjs` is NOT in Dockerfile.relay (verified via
grep), so no COPY-line update needed.

## Verification

- 14/14 power-reliability content-age tests pass
- 46/46 across Sprint 1 + Sprint 4 stack
- typecheck:api clean; lint clean
- Tests include a dedicated `fresh-arrival regression guard` test
  that pins the EXACT budget/natural-lag mismatch failure mode
  (Sprint 3b lesson made concrete) so a future budget tightening
  cannot silently re-introduce the immediate-page bug
- Boundary test: 2023 data in May 2026 (~29mo) DOES trip — confirms
  the staleness clock works correctly past the budget threshold

* fix(power-reliability): bump budget 24mo→36mo to cover steady-state ceiling

Greptile P1 on PR koala73#3602: 24-month budget false-positives mid-cycle when
next-year data publishes legitimately late.

The math I missed in the initial commit: fresh-arrival lag (~17mo for WB
EG.ELC.LOSS.ZS) is the FLOOR but not the worst case. Once year N is in
cache, it stays there until year N+1 publishes — which can legitimately
take up to end-of-(N+1) + 18mo = end-of-N + 30mo under the documented
12-18 month publication-lag range. So cache age can reach 30 months
between publications WITHOUT any real upstream stall.

Corrected budget = 30mo steady-state ceiling + 6mo slack = 36 months
(36 thirty-day-months = 1080 days ≈ 3 years).

Also resolves the P2 prose-vs-math mismatch (JSDoc previously said
"730 days" but `24 * 30 * 24 * 60` = 720; new wording "36 thirty-day
months ≈ 1080 days" is internally consistent).

General formula now documented in the helper JSDoc:

  budget >= max_publication_lag + cycle_length + slack

Both halves required: fresh-arrival lag AND cycle_length. Initial PR
covered fresh-arrival (~17mo) but missed cycle_length (12mo), which is
exactly how the false-positive emerges. Same shape as Sprint 3b PR koala73#3599
P1 — that one missed fresh-arrival; this one missed steady-state.

Tests:
- Renamed boundary test "max year 2023 (~29mo) DOES trip" → "steady-state
  regression guard: max year 2023 (~29mo) does NOT trip — within ceiling"
  with assertion direction flipped (29mo < 30mo ceiling = legitimate
  late-publication wait, not staleness)
- Added new boundary test "max year 2022 (~40mo) DOES trip — past ceiling
  = real stall" to confirm the budget fires correctly past the ceiling
- Constant assertion: 36 * 30 * 24 * 60

15/15 power-reliability tests pass; 47/47 across Sprint 1+4 stack;
typecheck:api clean; lint clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant