Skip to content

feat(disease-outbreaks): Sprint 2 — content-age pilot (synthetic tagging + 9d budget)#3597

Merged
koala73 merged 3 commits into
mainfrom
feat/health-readiness-content-age-sprint-2-disease-pilot
May 8, 2026
Merged

feat(disease-outbreaks): Sprint 2 — content-age pilot (synthetic tagging + 9d budget)#3597
koala73 merged 3 commits into
mainfrom
feat/health-readiness-content-age-sprint-2-disease-pilot

Conversation

@koala73

@koala73 koala73 commented May 4, 2026

Copy link
Copy Markdown
Owner

Stacked on PR #3596 (Sprint 1 — content-age probe infra). Merge #3596 first; this PR rebases onto main automatically once that lands.

Implements Sprint 2 of the 2026-05-04 health-readiness plan. Migrates disease-outbreaks as the proof-of-concept content-age consumer. Pilot maxContentAgeMin = 9 days chosen so the 2026-05-04 incident (cache held 50 outbreaks all 11+ days old, /api/health reported OK, map went empty) would have correctly tripped STALE_CONTENT.

Changes (scripts/seed-disease-outbreaks.mjs)

1. Synthetic-timestamp tagging across all 3 sources

Parser Existing fallback Migration
fetchWhoDonApi (line ~117) Date.now() when PublicationDateAndTime missing Carry _originalPublishedMs (parsed ms or null) + _publishedAtIsSynthetic (boolean). publishedMs keeps Date.now() fallback for UI consumer compat.
fetchRssItems (line ~150, both CDC and Outbreak News Today) Date.now() when pubDate missing Same pattern
fetchThinkGlobalHealth (line ~211) line-198 filter rejects undated already Always _publishedAtIsSynthetic: false, _originalPublishedMs: publishedMs (uniform shape across sources)

mapItem (line ~244) carries the helpers through to the output shape so contentMeta can read them at runSeed time.

2. runSeed contract opts

  • contentMeta: excludes _publishedAtIsSynthetic items + 1h clock-skew tolerance + null when validCount === 0. Synthetic-timestamp items would otherwise falsely report content as fresh and mask real upstream silence.
  • maxContentAgeMin: 9 * 24 * 60 = 12960 (9 days). Tighter would page on normal WHO/CDC quiet weeks; looser would have missed the 11d incident.
  • publishTransform: strips _publishedAtIsSynthetic + _originalPublishedMs from every item BEFORE atomicPublish. Per Sprint 1's ordering contract, contentMeta has already run on the raw data by this point — safe to drop. The strip ensures helpers never reach the canonical key, /api/bootstrap response, list-disease-outbreaks RPC response, or the proto-generated type.

Tests (tests/disease-outbreaks-seed.test.mjs — NEW)

16 cases split by layer per Codex round 4-5 review (pre-publish in-memory CARRIES helpers; post-strip canonical is helper-free):

Layer Cases
Pre-publish (parser) WHO/RSS without date → synthetic; WHO/RSS with date → non-synthetic; TGH always non-synthetic
contentMeta behavior All-synthetic → null; mixed → synthetic doesn't win newest; newest/oldest selection; future-dated >1h excluded; near-future ≤1h accepted
publishTransform strip Both fields stripped; publishedAt remains non-null; empty/missing handled
End-to-end Combined-regex assertion: published JSON contains NEITHER helper name
Pilot threshold 11d-old items DO trip; 5d-old items DO NOT (anti-drift)

Test totals: 95/95 pass across seed-envelope, seed-contract, seed-utils, content-age, health-content-age, and disease-outbreaks-seed suites.

Net diff

2 files changed, 449 insertions(+), 10 deletions(-)

Verification (post-deploy)

After Railway redeploy of seed-bundle-health:

  • /api/health.diseaseOutbreaks reports contentAgeMin and maxContentAgeMin: 12960
  • With current cache (50 items, newest ~11d old per 2026-05-04 incident), /api/health.diseaseOutbreaks.status === 'STALE_CONTENT'
  • Redis canonical helper-free:
    redis-cli GET health:disease-outbreaks:v1 | grep -cE '_publishedAtIsSynthetic|_originalPublishedMs'
    # Expected: 0
  • Same combined-regex check on /api/bootstrap?keys=diseaseOutbreaks response body returns 0
  • After TGH fix from fix(disease-outbreaks): TGH master URL + new bundle parser + skip time filter #3593 lands daily-fresh items, STALE_CONTENT clears naturally as newestItemAt shifts to within 9 days

What's next

Sprint 3 — sparse seeders (climate news, IEA OPEC, news-digest, economic stress).
Sprint 4 — annual-data seeders (WB resilience indicators).

@vercel

vercel Bot commented May 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment May 8, 2026 8:57am

Request Review

@greptile-apps

greptile-apps Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Implements Sprint 2 of the health-readiness content-age plan: adds _publishedAtIsSynthetic / _originalPublishedMs helper fields across all three disease-outbreak sources (WHO DON API, CDC/ONT RSS, ThinkGlobalHealth), wires them into a new contentMeta function with a 9-day staleness budget (maxContentAgeMin: 12960), and strips them via publishTransform before the canonical Redis key is written. The contentMeta → publishTransform ordering contract confirmed in _seed-utils.mjs is respected, all helper fields are correctly normalized by mapItem, and the validateDescriptor cross-field check ensures contentMeta and maxContentAgeMin opt-in together. The two P2 findings are minor: the test suite replicates parser logic locally rather than importing it (silent drift risk), and one test case has a timing sensitivity around the 1h clock-skew boundary.

Confidence Score: 4/5

Safe to merge; no data-correctness or ordering bugs found — only test quality P2s.

All P0/P1 surfaces checked: contentMeta runs before publishTransform (confirmed in _seed-utils.mjs), helpers are stripped correctly and never reach canonical key or API responses, null/undefined/Infinity edge cases in contentMeta are handled, and the maxContentAgeMin integer contract is satisfied. Two P2 findings (replicated test logic, timing sensitivity in one test) do not affect runtime correctness.

tests/disease-outbreaks-seed.test.mjs — test suite validates local replicas rather than actual seeder functions, creating a silent drift risk on future seeder changes.

Important Files Changed

Filename Overview
scripts/seed-disease-outbreaks.mjs Adds synthetic-timestamp tagging across all 3 sources (WHO/RSS/TGH) and wires contentMeta + publishTransform into runSeed opts; logic is correct and ordering contract (contentMeta before publishTransform) matches the confirmed _seed-utils.mjs implementation.
tests/disease-outbreaks-seed.test.mjs 16 new tests covering pre-publish helpers, contentMeta, publishTransform, and threshold sanity; tests replicate seeder parser logic inline instead of importing from the seeder, creating a silent drift risk if the real seeder logic changes without updating the test copies.

Sequence Diagram

sequenceDiagram
    participant F as fetchDiseaseOutbreaks
    participant P as Parsers (WHO/RSS/TGH)
    participant MI as mapItem
    participant CM as contentMeta
    participant PT as publishTransform
    participant RD as Redis (canonical key)
    participant HLT as /api/health

    F->>P: fetch raw items
    P-->>F: items with _publishedAtIsSynthetic + _originalPublishedMs
    F->>MI: map each item
    MI-->>F: normalized items (helpers forwarded)
    Note over F: data = { outbreaks, fetchedAt }

    F->>CM: contentMeta(data) [RAW - helpers present]
    CM-->>F: { newestItemAt, oldestItemAt } or null
    Note over CM: Skips synthetic items<br/>Skips ts > skewLimit (1h)<br/>Returns null → STALE_CONTENT

    F->>PT: publishTransform(data)
    PT-->>F: publishData (helpers stripped)

    F->>RD: atomicPublish(publishData) + envelope with newestItemAt/maxContentAgeMin
    RD-->>F: ACK

    HLT->>RD: read seed-meta
    RD-->>HLT: newestItemAt, maxContentAgeMin=12960
    HLT-->>HLT: ageMin > 12960 → STALE_CONTENT
Loading

Reviews (1): Last reviewed commit: "feat(disease-outbreaks): Sprint 2 — cont..." | Re-trigger Greptile

Comment thread tests/disease-outbreaks-seed.test.mjs Outdated
Comment thread tests/disease-outbreaks-seed.test.mjs
koala73 added a commit that referenced this pull request May 4, 2026
…est drift and timing flake

Greptile P2s on PR #3597:

1. tests/disease-outbreaks-seed.test.mjs replicated parser/mapper/contentMeta
   logic locally — a drift in fetchWhoDonApi or contentMeta would not have
   failed any of the 16 tests because they asserted against their own copy
   of the logic, not the seeder's.

2. The "near-future ≤1h accepted" test relied on Date.now() being stable
   between test setup and the call into contentMeta. On a loaded CI runner
   the gap could exceed the (1h - 30min) margin and flake.

Fixes both at once:

- New scripts/_disease-outbreaks-helpers.mjs exports the pure functions
  (whoNormalizeItem, rssNormalizeItem, tghNormalizeItem, mapItem,
  diseaseContentMeta, diseasePublishTransform, DISEASE_MAX_CONTENT_AGE_MIN).
  diseaseContentMeta accepts an optional nowMs for deterministic skew tests.

- Seeder imports those helpers instead of inlining them. ~150 lines
  removed; behavior unchanged (verified by node -c + smoke test).

- Test file imports the real helpers (no replicas). All skew-limit tests
  inject FIXED_NOW=1700000000000 — no wall-clock dependence.

- Tightens the "within 1h tolerance" test from +30min to +5min ahead of
  injected NOW, well clear of the 1h boundary regardless of the timing fix.

Net: -265 lines across the two existing files; +200 in the new helpers
module. 17/17 disease tests pass; 49/49 across the full Sprint 1+2 stack.
Base automatically changed from feat/health-readiness-content-age-sprint-1 to main May 8, 2026 08:43
koala73 added 3 commits May 8, 2026 12:53
…ing + STALE_CONTENT @ 9d)

Implements Sprint 2 of the 2026-05-04 health-readiness plan
(docs/plans/2026-05-04-001-feat-health-readiness-probe-content-age-plan.md).
Stacked on Sprint 1 (#3596 — content-age probe infra).

Migrates disease-outbreaks as the proof-of-concept content-age consumer.
Pilot maxContentAgeMin=9 days chosen so the 2026-05-04 11d-old incident
would have correctly tripped STALE_CONTENT.

== Source-parser changes (3 sources, uniform shape) ==

scripts/seed-disease-outbreaks.mjs:

WHO DON parser (line ~117): tag synthetic timestamps when the upstream
omits PublicationDateAndTime. Carry _originalPublishedMs (parsed ms or
null) and _publishedAtIsSynthetic (boolean) alongside the existing
publishedMs (which keeps its Date.now() fallback for UI consumer compat).

RSS parser (line ~150, both CDC and Outbreak News Today): same pattern
when pubDate is missing/unparseable.

TGH parser (line ~211): always carries non-synthetic since the line-198
filter rejects undated items earlier. Migration is additive — every
TGH item gets _publishedAtIsSynthetic: false and _originalPublishedMs:
publishedMs so contentMeta + publishTransform apply uniformly.

mapItem (line ~244): carries _publishedAtIsSynthetic and
_originalPublishedMs through to the output shape so contentMeta can
read them at runSeed time.

== runSeed opts (Sprint 2 contract) ==

contentMeta: excludes _publishedAtIsSynthetic items + 1h clock-skew
tolerance + null when validCount === 0 (matches list-feed-digest's
FUTURE_DATE_TOLERANCE_MS pattern).

maxContentAgeMin: 9 * 24 * 60 = 12960 minutes (9 days) — chosen
deliberately so the production incident's 11d-old cache would have
flagged STALE_CONTENT. Tighter would page on normal WHO/CDC quiet
weeks; looser would have missed the incident.

publishTransform: strips _publishedAtIsSynthetic + _originalPublishedMs
from every item BEFORE atomicPublish so the helpers never reach:
  - the Redis canonical key (health:disease-outbreaks:v1)
  - /api/bootstrap response (data.diseaseOutbreaks)
  - list-disease-outbreaks RPC response
  - the DiseaseOutbreakItem proto-generated type

The Sprint 1 ordering contract (contentMeta runs BEFORE publishTransform)
guarantees contentMeta sees the helpers that publishTransform then strips.

== Anti-regression tests ==

tests/disease-outbreaks-seed.test.mjs (NEW) — 16 cases split by layer:

Pre-publish (in-memory) layer (5):
- WHO without PublicationDateAndTime → tagged synthetic
- WHO with valid PublicationDateAndTime → non-synthetic
- RSS without pubDate → tagged synthetic
- RSS with valid pubDate → non-synthetic
- TGH always non-synthetic

contentMeta behavior (5):
- All-synthetic → null (→ STALE_CONTENT)
- Mixed: synthetic with newer publishedAt does NOT win newest
- Picks newest+oldest from non-synthetic set
- Future-dated items beyond 1h tolerance excluded
- NEAR_FUTURE within 1h tolerance accepted

publishTransform strip (3):
- Both helper fields stripped from every item
- publishedAt remains non-null (UI/RPC consumer contract)
- Empty + missing outbreaks handled safely

End-to-end (1):
- contentMeta runs on raw data WITH helpers, publishTransform strips,
  canonical-shape JSON contains NEITHER _publishedAtIsSynthetic NOR
  _originalPublishedMs (combined-regex assertion per Codex round 4 P2)

Pilot threshold sanity (2):
- 11d-old items DO trip the 9d budget (anti-drift on the pilot threshold —
  any future change to 9d must update this test)
- 5d-old items DO NOT trip (no false positive on normal upstream rhythm)

Test totals: 95/95 pass across the seed-envelope, seed-contract,
seed-utils, content-age, health-content-age, and disease-outbreaks-seed
suites.

== Verification (post-deploy) ==

After Railway bundle redeploy:
1. /api/health.diseaseOutbreaks shows contentAgeMin and maxContentAgeMin.
2. Redis canonical health:disease-outbreaks:v1 contains NEITHER
   _publishedAtIsSynthetic NOR _originalPublishedMs (combined-regex
   grep returns 0).
3. /api/bootstrap?keys=diseaseOutbreaks response payload helper-free.
4. With current 11d-old WHO/CDC items + bug-pattern data, STALE_CONTENT
   surfaces in /api/health and ops can act on it.
…est drift and timing flake

Greptile P2s on PR #3597:

1. tests/disease-outbreaks-seed.test.mjs replicated parser/mapper/contentMeta
   logic locally — a drift in fetchWhoDonApi or contentMeta would not have
   failed any of the 16 tests because they asserted against their own copy
   of the logic, not the seeder's.

2. The "near-future ≤1h accepted" test relied on Date.now() being stable
   between test setup and the call into contentMeta. On a loaded CI runner
   the gap could exceed the (1h - 30min) margin and flake.

Fixes both at once:

- New scripts/_disease-outbreaks-helpers.mjs exports the pure functions
  (whoNormalizeItem, rssNormalizeItem, tghNormalizeItem, mapItem,
  diseaseContentMeta, diseasePublishTransform, DISEASE_MAX_CONTENT_AGE_MIN).
  diseaseContentMeta accepts an optional nowMs for deterministic skew tests.

- Seeder imports those helpers instead of inlining them. ~150 lines
  removed; behavior unchanged (verified by node -c + smoke test).

- Test file imports the real helpers (no replicas). All skew-limit tests
  inject FIXED_NOW=1700000000000 — no wall-clock dependence.

- Tightens the "within 1h tolerance" test from +30min to +5min ahead of
  injected NOW, well clear of the 1h boundary regardless of the timing fix.

Net: -265 lines across the two existing files; +200 in the new helpers
module. 17/17 disease tests pass; 49/49 across the full Sprint 1+2 stack.
Unix timestamp 1700000000000 ms is 2023-11-14T22:13:20Z, not 2025-11-14.
Test correctness unaffected (FIXED_NOW is just an injected stable epoch),
but a reader reasoning about the skew-limit arithmetic would get the
mental date math wrong. Greptile P2 on PR #3598 (which copied the same
wrong comment from this file when Sprint 3a was branched off).
@koala73
koala73 force-pushed the feat/health-readiness-content-age-sprint-2-disease-pilot branch from 1bcf7c5 to 287e3e6 Compare May 8, 2026 08:54
koala73 added a commit that referenced this pull request May 8, 2026
Sparse seeders sub-PR b/c of the 2026-05-04 health-readiness plan.
Branched off Sprint 1 (#3596) as a parallel sibling to Sprint 2 (#3597)
and Sprint 3a (#3598) per the plan's "Each PR is independently shippable"
note (line 498).

## Why this matters

IEA monthly oil stocks publish on an M+2 cadence — August data ships in
late October/early November. Without a content-age probe, a stalled
publication month is invisible to /api/health: the seeder runs fine on
its 6h cron, fetchedAt stays fresh, but data.dataMonth never advances. A
45-day budget trips STALE_CONTENT exactly when a month has been missed
(e.g. cache shows "2024-08" past Dec 1 when "2024-10" should have
landed).

## Shape contract — different from Sprint 2/3a

IEA is a SINGLE-SNAPSHOT seeder: every member shares one `dataMonth`
("YYYY-MM" string at the top level), there is no per-item published-at.
The new helper parses dataMonth → end-of-month UTC ms (the latest
possible observation date in the named period) and returns it as both
`newestItemAt` and `oldestItemAt`.

Defensive: contentMeta returns null when dataMonth is missing, malformed
("2024-13", "2024-8" single-digit), or future-dated beyond 1h clock-skew
tolerance (guards against upstream yearMonth garbage producing e.g. a
2099-12 dataMonth).

## Pattern parity with Sprint 2/3a

Following the established pattern: pure helpers in
`scripts/_iea-oil-stocks-helpers.mjs` (`dataMonthToEndOfMonthMs`,
`ieaOilStocksContentMeta`, `IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN`).
Seeder imports them; tests import them. No replicas.

`seed-iea-oil-stocks.mjs` is NOT in Dockerfile.relay (verified via
`grep`), so no COPY-line update needed (unlike Sprint 3a's
seed-climate-news which IS relay-COPY'd).

## Verification

- 15/15 iea content-age tests pass (incl. leap-year, month-rollover,
  invalid-shape rejection, M+2 lag realism, future-clock-skew defense)
- 78/78 across iea seed + Sprint 1 + Sprint 3b stack
- typecheck:api clean; lint clean (pre-existing warnings only)
- Dockerfile.relay closure test passes (no relay impact)
@koala73
koala73 merged commit 134e1d7 into main May 8, 2026
11 checks passed
@koala73
koala73 deleted the feat/health-readiness-content-age-sprint-2-disease-pilot branch May 8, 2026 09:00
koala73 added a commit that referenced this pull request May 8, 2026
* feat(iea-oil-stocks): Sprint 3b — content-age probe (45d budget)

Sparse seeders sub-PR b/c of the 2026-05-04 health-readiness plan.
Branched off Sprint 1 (#3596) as a parallel sibling to Sprint 2 (#3597)
and Sprint 3a (#3598) per the plan's "Each PR is independently shippable"
note (line 498).

## Why this matters

IEA monthly oil stocks publish on an M+2 cadence — August data ships in
late October/early November. Without a content-age probe, a stalled
publication month is invisible to /api/health: the seeder runs fine on
its 6h cron, fetchedAt stays fresh, but data.dataMonth never advances. A
45-day budget trips STALE_CONTENT exactly when a month has been missed
(e.g. cache shows "2024-08" past Dec 1 when "2024-10" should have
landed).

## Shape contract — different from Sprint 2/3a

IEA is a SINGLE-SNAPSHOT seeder: every member shares one `dataMonth`
("YYYY-MM" string at the top level), there is no per-item published-at.
The new helper parses dataMonth → end-of-month UTC ms (the latest
possible observation date in the named period) and returns it as both
`newestItemAt` and `oldestItemAt`.

Defensive: contentMeta returns null when dataMonth is missing, malformed
("2024-13", "2024-8" single-digit), or future-dated beyond 1h clock-skew
tolerance (guards against upstream yearMonth garbage producing e.g. a
2099-12 dataMonth).

## Pattern parity with Sprint 2/3a

Following the established pattern: pure helpers in
`scripts/_iea-oil-stocks-helpers.mjs` (`dataMonthToEndOfMonthMs`,
`ieaOilStocksContentMeta`, `IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN`).
Seeder imports them; tests import them. No replicas.

`seed-iea-oil-stocks.mjs` is NOT in Dockerfile.relay (verified via
`grep`), so no COPY-line update needed (unlike Sprint 3a's
seed-climate-news which IS relay-COPY'd).

## Verification

- 15/15 iea content-age tests pass (incl. leap-year, month-rollover,
  invalid-shape rejection, M+2 lag realism, future-clock-skew defense)
- 78/78 across iea seed + Sprint 1 + Sprint 3b stack
- typecheck:api clean; lint clean (pre-existing warnings only)
- Dockerfile.relay closure test passes (no relay impact)

* fix(iea-oil-stocks): bump budget 45d→90d to cover M+2 natural lag

Greptile P1 on PR #3599: a 45-day budget contradicts the helper's own
M+2 cadence claim. End-of-observation-month (Aug 31) is ~60-65 days
BEFORE publication (~late Oct/early Nov), so fresh-arrival data is
already past the 45d threshold at the moment a successful seed run
writes it. STALE_CONTENT would have fired on every cron tick.

Corrected math: 90d = ~60d natural M+2 lag + ~30d missed-publication
slack. Trips only when a month is missed entirely (cache stuck at
"2024-08" past mid-Jan when "2024-10" should have landed).

Also addresses 3 P2 review nits in the same edit:

- Test "60 days old" → "fresh-arrival regression guard: ~60d-old fresh
  M+2 data does NOT trip" (the math was right, name was wrong; rewrote
  the test to actually pin the failure mode the P1 cited).
- Test "~30 days old" → "~14 days old" (the fixture was "2023-10" =
  ~14d before FIXED_NOW, not 30).
- M+2 lag scenario comment "Sept data published ~Oct 25" → "~late Nov
  (M+2 cadence)" — Oct 25 is M+1, not M+2.

Added: dedicated fresh-arrival regression guard test that asserts a
~75d-old fresh M+2 dataMonth is within budget. Without it, a future
budget tightening could re-introduce the immediate-page bug invisibly.

Verification: 16/16 iea content-age (was 15/15 — added regression guard);
79/79 across iea seed + Sprint 1 + Sprint 3b stack; typecheck:api clean.
fuleinist pushed a commit to fuleinist/worldmonitor that referenced this pull request May 9, 2026
…ing + 9d budget) (koala73#3597)

* feat(disease-outbreaks): Sprint 2 — content-age pilot (synthetic tagging + STALE_CONTENT @ 9d)

Implements Sprint 2 of the 2026-05-04 health-readiness plan
(docs/plans/2026-05-04-001-feat-health-readiness-probe-content-age-plan.md).
Stacked on Sprint 1 (koala73#3596 — content-age probe infra).

Migrates disease-outbreaks as the proof-of-concept content-age consumer.
Pilot maxContentAgeMin=9 days chosen so the 2026-05-04 11d-old incident
would have correctly tripped STALE_CONTENT.

== Source-parser changes (3 sources, uniform shape) ==

scripts/seed-disease-outbreaks.mjs:

WHO DON parser (line ~117): tag synthetic timestamps when the upstream
omits PublicationDateAndTime. Carry _originalPublishedMs (parsed ms or
null) and _publishedAtIsSynthetic (boolean) alongside the existing
publishedMs (which keeps its Date.now() fallback for UI consumer compat).

RSS parser (line ~150, both CDC and Outbreak News Today): same pattern
when pubDate is missing/unparseable.

TGH parser (line ~211): always carries non-synthetic since the line-198
filter rejects undated items earlier. Migration is additive — every
TGH item gets _publishedAtIsSynthetic: false and _originalPublishedMs:
publishedMs so contentMeta + publishTransform apply uniformly.

mapItem (line ~244): carries _publishedAtIsSynthetic and
_originalPublishedMs through to the output shape so contentMeta can
read them at runSeed time.

== runSeed opts (Sprint 2 contract) ==

contentMeta: excludes _publishedAtIsSynthetic items + 1h clock-skew
tolerance + null when validCount === 0 (matches list-feed-digest's
FUTURE_DATE_TOLERANCE_MS pattern).

maxContentAgeMin: 9 * 24 * 60 = 12960 minutes (9 days) — chosen
deliberately so the production incident's 11d-old cache would have
flagged STALE_CONTENT. Tighter would page on normal WHO/CDC quiet
weeks; looser would have missed the incident.

publishTransform: strips _publishedAtIsSynthetic + _originalPublishedMs
from every item BEFORE atomicPublish so the helpers never reach:
  - the Redis canonical key (health:disease-outbreaks:v1)
  - /api/bootstrap response (data.diseaseOutbreaks)
  - list-disease-outbreaks RPC response
  - the DiseaseOutbreakItem proto-generated type

The Sprint 1 ordering contract (contentMeta runs BEFORE publishTransform)
guarantees contentMeta sees the helpers that publishTransform then strips.

== Anti-regression tests ==

tests/disease-outbreaks-seed.test.mjs (NEW) — 16 cases split by layer:

Pre-publish (in-memory) layer (5):
- WHO without PublicationDateAndTime → tagged synthetic
- WHO with valid PublicationDateAndTime → non-synthetic
- RSS without pubDate → tagged synthetic
- RSS with valid pubDate → non-synthetic
- TGH always non-synthetic

contentMeta behavior (5):
- All-synthetic → null (→ STALE_CONTENT)
- Mixed: synthetic with newer publishedAt does NOT win newest
- Picks newest+oldest from non-synthetic set
- Future-dated items beyond 1h tolerance excluded
- NEAR_FUTURE within 1h tolerance accepted

publishTransform strip (3):
- Both helper fields stripped from every item
- publishedAt remains non-null (UI/RPC consumer contract)
- Empty + missing outbreaks handled safely

End-to-end (1):
- contentMeta runs on raw data WITH helpers, publishTransform strips,
  canonical-shape JSON contains NEITHER _publishedAtIsSynthetic NOR
  _originalPublishedMs (combined-regex assertion per Codex round 4 P2)

Pilot threshold sanity (2):
- 11d-old items DO trip the 9d budget (anti-drift on the pilot threshold —
  any future change to 9d must update this test)
- 5d-old items DO NOT trip (no false positive on normal upstream rhythm)

Test totals: 95/95 pass across the seed-envelope, seed-contract,
seed-utils, content-age, health-content-age, and disease-outbreaks-seed
suites.

== Verification (post-deploy) ==

After Railway bundle redeploy:
1. /api/health.diseaseOutbreaks shows contentAgeMin and maxContentAgeMin.
2. Redis canonical health:disease-outbreaks:v1 contains NEITHER
   _publishedAtIsSynthetic NOR _originalPublishedMs (combined-regex
   grep returns 0).
3. /api/bootstrap?keys=diseaseOutbreaks response payload helper-free.
4. With current 11d-old WHO/CDC items + bug-pattern data, STALE_CONTENT
   surfaces in /api/health and ops can act on it.

* refactor(disease-outbreaks): extract helpers + inject nowMs to kill test drift and timing flake

Greptile P2s on PR koala73#3597:

1. tests/disease-outbreaks-seed.test.mjs replicated parser/mapper/contentMeta
   logic locally — a drift in fetchWhoDonApi or contentMeta would not have
   failed any of the 16 tests because they asserted against their own copy
   of the logic, not the seeder's.

2. The "near-future ≤1h accepted" test relied on Date.now() being stable
   between test setup and the call into contentMeta. On a loaded CI runner
   the gap could exceed the (1h - 30min) margin and flake.

Fixes both at once:

- New scripts/_disease-outbreaks-helpers.mjs exports the pure functions
  (whoNormalizeItem, rssNormalizeItem, tghNormalizeItem, mapItem,
  diseaseContentMeta, diseasePublishTransform, DISEASE_MAX_CONTENT_AGE_MIN).
  diseaseContentMeta accepts an optional nowMs for deterministic skew tests.

- Seeder imports those helpers instead of inlining them. ~150 lines
  removed; behavior unchanged (verified by node -c + smoke test).

- Test file imports the real helpers (no replicas). All skew-limit tests
  inject FIXED_NOW=1700000000000 — no wall-clock dependence.

- Tightens the "within 1h tolerance" test from +30min to +5min ahead of
  injected NOW, well clear of the 1h boundary regardless of the timing fix.

Net: -265 lines across the two existing files; +200 in the new helpers
module. 17/17 disease tests pass; 49/49 across the full Sprint 1+2 stack.

* fix(test): correct FIXED_NOW comment year (2025→2023)

Unix timestamp 1700000000000 ms is 2023-11-14T22:13:20Z, not 2025-11-14.
Test correctness unaffected (FIXED_NOW is just an injected stable epoch),
but a reader reasoning about the skew-limit arithmetic would get the
mental date math wrong. Greptile P2 on PR koala73#3598 (which copied the same
wrong comment from this file when Sprint 3a was branched off).
fuleinist pushed a commit to fuleinist/worldmonitor that referenced this pull request May 9, 2026
…la73#3599)

* feat(iea-oil-stocks): Sprint 3b — content-age probe (45d budget)

Sparse seeders sub-PR b/c of the 2026-05-04 health-readiness plan.
Branched off Sprint 1 (koala73#3596) as a parallel sibling to Sprint 2 (koala73#3597)
and Sprint 3a (koala73#3598) per the plan's "Each PR is independently shippable"
note (line 498).

## Why this matters

IEA monthly oil stocks publish on an M+2 cadence — August data ships in
late October/early November. Without a content-age probe, a stalled
publication month is invisible to /api/health: the seeder runs fine on
its 6h cron, fetchedAt stays fresh, but data.dataMonth never advances. A
45-day budget trips STALE_CONTENT exactly when a month has been missed
(e.g. cache shows "2024-08" past Dec 1 when "2024-10" should have
landed).

## Shape contract — different from Sprint 2/3a

IEA is a SINGLE-SNAPSHOT seeder: every member shares one `dataMonth`
("YYYY-MM" string at the top level), there is no per-item published-at.
The new helper parses dataMonth → end-of-month UTC ms (the latest
possible observation date in the named period) and returns it as both
`newestItemAt` and `oldestItemAt`.

Defensive: contentMeta returns null when dataMonth is missing, malformed
("2024-13", "2024-8" single-digit), or future-dated beyond 1h clock-skew
tolerance (guards against upstream yearMonth garbage producing e.g. a
2099-12 dataMonth).

## Pattern parity with Sprint 2/3a

Following the established pattern: pure helpers in
`scripts/_iea-oil-stocks-helpers.mjs` (`dataMonthToEndOfMonthMs`,
`ieaOilStocksContentMeta`, `IEA_OIL_STOCKS_MAX_CONTENT_AGE_MIN`).
Seeder imports them; tests import them. No replicas.

`seed-iea-oil-stocks.mjs` is NOT in Dockerfile.relay (verified via
`grep`), so no COPY-line update needed (unlike Sprint 3a's
seed-climate-news which IS relay-COPY'd).

## Verification

- 15/15 iea content-age tests pass (incl. leap-year, month-rollover,
  invalid-shape rejection, M+2 lag realism, future-clock-skew defense)
- 78/78 across iea seed + Sprint 1 + Sprint 3b stack
- typecheck:api clean; lint clean (pre-existing warnings only)
- Dockerfile.relay closure test passes (no relay impact)

* fix(iea-oil-stocks): bump budget 45d→90d to cover M+2 natural lag

Greptile P1 on PR koala73#3599: a 45-day budget contradicts the helper's own
M+2 cadence claim. End-of-observation-month (Aug 31) is ~60-65 days
BEFORE publication (~late Oct/early Nov), so fresh-arrival data is
already past the 45d threshold at the moment a successful seed run
writes it. STALE_CONTENT would have fired on every cron tick.

Corrected math: 90d = ~60d natural M+2 lag + ~30d missed-publication
slack. Trips only when a month is missed entirely (cache stuck at
"2024-08" past mid-Jan when "2024-10" should have landed).

Also addresses 3 P2 review nits in the same edit:

- Test "60 days old" → "fresh-arrival regression guard: ~60d-old fresh
  M+2 data does NOT trip" (the math was right, name was wrong; rewrote
  the test to actually pin the failure mode the P1 cited).
- Test "~30 days old" → "~14 days old" (the fixture was "2023-10" =
  ~14d before FIXED_NOW, not 30).
- M+2 lag scenario comment "Sept data published ~Oct 25" → "~late Nov
  (M+2 cadence)" — Oct 25 is M+1, not M+2.

Added: dedicated fresh-arrival regression guard test that asserts a
~75d-old fresh M+2 dataMonth is within budget. Without it, a future
budget tightening could re-introduce the immediate-page bug invisibly.

Verification: 16/16 iea content-age (was 15/15 — added regression guard);
79/79 across iea seed + Sprint 1 + Sprint 3b stack; typecheck:api clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant