fix(disease-outbreaks): TGH master URL + new bundle parser + skip time filter by koala73 · Pull Request #3593 · koala73/worldmonitor

koala73 · 2026-05-04T16:59:42Z

Production 2026-05-04: enabling the disease-outbreaks map layer renders nothing despite /api/health reporting diseaseOutbreaks: status=OK, records=50. Direct Redis read confirmed cache holds 50 items. Three compound issues, fixed in one PR.

A. ThinkGlobalHealth source returning 0 items

Two sub-bugs in fetchThinkGlobalHealth:

Wrong default branch: hardcoded /main/index_bundle.js but TGH repo's default is master. The /main/ URL has been silent 404 — !resp.ok caught and returned []. Fix: main → master. Verified live: 200 OK, 7.5 MB bundle.
Bundle format change after 2026-04 webpack rebuild: legacy parser anchored on var a=[{Alert_ID: (unquoted JS keys). New bundle wraps records in eval("var res = [...]") blocks with JSON-quoted keys. Old regex never matches the quoted-key form. Fix: import parseRealtimeAlerts from seed-vpd-tracker.mjs (which already handles this exact format with a battle-tested schema-anchored scanner — VPD and Disease share the same TGH bundle).

After both fixes TGH contributes ~1,600 ProMED-reviewed alerts with real lat/lng — restoring the only geo-rich source.

B. UI time filter eats every item

DeckGLMap.ts:1505 gated diseaseOutbreaks by the global timeRange dropdown (max '7d'). Disease outbreaks are sparse-by-nature — WHO DON publishes 1-2/week, CDC HAN alerts are infrequent. When the most recent WHO/CDC update is 8+ days old (normal), the 7d gate dropped every item → empty layer. Production confirmed: 50 cached items, newest 11.0d old, all dropped.

Fix: skip the time filter for diseaseOutbreaks. Render all items in the cache; the seeder's per-source lookback already bounds freshness at write time. Other layers keep the global filter.

Out of scope

C: structural health-readiness probe (seedAgeMin tracks seeder run, not item freshness — separate followup).
Static-layer zoom-gates (bases/nuclear/spaceports/economic show nothing at default zoom 2 because LAYER_ZOOM_THRESHOLDS[*].minZoom is 3-5). Intentional UX, not a data bug — separate followup if we want a "zoom in to see N items" affordance.

Test plan

node -c on seeder + npm run typecheck clean
After deploy + Railway bundle redeploy: TGH source contributes ~1,600 records → 50+ cached items have real lat/lng
Map's disease-outbreaks layer renders all cached items regardless of timeRange

…kip time filter PRODUCTION 2026-05-04: enabling the disease-outbreaks map layer renders nothing despite /api/health reporting `diseaseOutbreaks: status=OK, records=50`. Direct Redis read confirmed the cache holds 50 items but none render. Three compound issues, fixed in one PR. == A. ThinkGlobalHealth source returning 0 items == Two sub-bugs in `fetchThinkGlobalHealth`: 1. Wrong default branch in URL. The seeder hardcoded `/main/index_bundle.js` but the TGH GitHub repo's default branch is `master`. The `/main/` URL has been returning HTTP 404 silently — the seeder caught the !resp.ok and returned []. Fix: `main` → `master`. Verified live: 200 OK, 7.5 MB bundle. 2. Bundle format change after 2026-04 webpack rebuild. The legacy parser anchored on `var a=[{Alert_ID:` (unquoted JS keys). The new bundle wraps records in `eval("var res = [...]")` blocks with JSON-quoted keys like `\"Alert_ID\":\"8732529\"`. The old regex (`/(\w+):"((?:[^"\\]|\\.)*)"/`) doesn't match the quoted-key form. Fix: import `parseRealtimeAlerts` from seed-vpd-tracker.mjs (which already handles this exact format with a battle-tested schema-anchored scanner — VPD and Disease share the same TGH bundle). After both fixes, TGH contributes ~1,600 ProMED-reviewed alerts with real lat/lng, restoring the only geo-rich source (WHO/CDC/ONT only publish country names → require centroid fallback). == B. UI time filter eats every item == The map's `filterByTimeCached` (DeckGLMap.ts:1505) gated diseaseOutbreaks by the global timeRange dropdown (max '7d'). Disease outbreaks are sparse-by-nature — WHO DON publishes 1-2/week, CDC HAN alerts are infrequent, TGH carries 90 days of ProMED items. When the most recent WHO/CDC update is 8+ days old (normal), the 7d gate dropped every item → empty layer. Production confirmed: 50 cached items, newest 11.0 days old, all dropped. Fix: skip the time filter for diseaseOutbreaks. Render all items in the cache; the seeder's per-source lookback already bounds freshness at write time. Other layers keep the global filter. == Out of scope == - C: structural health-readiness probe (seedAgeMin tracks seeder run, not item freshness — separate followup). - Static-layer zoom-gates (bases/nuclear/spaceports/economic show nothing at default zoom 2 because LAYER_ZOOM_THRESHOLDS[*].minZoom is 3-5). Intentional UX, not a data bug — separate followup if we want a "zoom in to see N items" affordance.

vercel · 2026-05-04T16:59:46Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	May 4, 2026 5:02pm

greptile-apps · 2026-05-04T17:03:47Z

Greptile Summary

This PR fixes three compounding issues that caused the disease-outbreaks map layer to render nothing in production: the TGH GitHub URL is corrected from main to master, the broken homegrown bundle parser is replaced with the battle-tested parseRealtimeAlerts from seed-vpd-tracker.mjs (which handles the post-2026-04 webpack format), and the global time-range filter is bypassed for the disease-outbreaks layer to prevent sparse sources from being dropped entirely. All field mappings between the imported parser's output schema and the seeder's item builder are consistent.

Confidence Score: 4/5

Safe to merge — all three fixes are well-scoped and the only findings are P2 style/UX concerns.

No P0 or P1 issues found. The parser import is clean, field mappings are verified against the upstream function, and the time-filter bypass is intentional and documented. Two P2 items remain: a JSDoc field-name mismatch and the silent divergence from the global time-filter contract in the map UI.

No files require special attention beyond the noted P2 items.

Important Files Changed

Filename	Overview
scripts/seed-disease-outbreaks.mjs	Fixes the TGH source URL (main→master) and replaces the homegrown bundle parser with `parseRealtimeAlerts` from `seed-vpd-tracker.mjs`; field mappings (`disease`, `placeName`, `sourceUrl`, numeric `lat`/`lng`) are all consistent with the imported parser's output schema. Minor: JSDoc in `fetchThinkGlobalHealth` references the raw bundle field `diseases` (plural) rather than the remapped `disease` (singular) that callers actually use.
src/components/DeckGLMap.ts	Removes the `filterByTimeCached` call for `diseaseOutbreaks`, making the layer immune to the global time-range dropdown. The motivation is well-documented, but this creates a silent asymmetry with every other filtered layer and no UI feedback to users.

Sequence Diagram

sequenceDiagram
    participant Cron as Railway Cron
    participant DS as seed-disease-outbreaks.mjs
    participant VPD as seed-vpd-tracker.mjs (parseRealtimeAlerts)
    participant TGH as raw.githubusercontent.com/master/index_bundle.js
    participant WHO as WHO DON API
    participant CDC as CDC RSS
    participant Redis as Redis Cache

    Cron->>DS: run seeder
    DS->>TGH: fetch bundle (master branch ✓)
    TGH-->>DS: 7.5 MB JS bundle
    DS->>VPD: parseRealtimeAlerts(bundle)
    VPD-->>DS: ~1,600 normalised alerts (disease, lat, lng, …)
    DS->>DS: filter to last 90d
    DS->>WHO: fetch DON API
    WHO-->>DS: recent outbreak notices
    DS->>CDC: fetch RSS
    CDC-->>DS: HAN alerts
    DS->>DS: deduplicate + cap (150 TGH + 50 others)
    DS->>Redis: write health:disease-outbreaks:v1 (TTL 72h)

    Note over DS,Redis: UI reads from cache — time filter skipped for this layer

_{Reviews (1): Last reviewed commit: "fix(disease-outbreaks): TGH master-branc..." | Re-trigger Greptile}

greptile-apps · 2026-05-04T17:03:51Z

+ * as the field names (Alert_ID, lat, lng, diseases) are stable.
+ *
+ * We filter to last TGH_LOOKBACK_DAYS days post-parse.


JSDoc field name mismatch (diseases vs disease)

The comment says "as long as the field names (Alert_ID, lat, lng, diseases) are stable," but parseRealtimeAlerts maps r.diseases → .disease (singular) before returning. Callers therefore work with rec.disease (which this function correctly uses), but the JSDoc implies the stable name is the raw diseases key. A future maintainer debugging a field-not-found error would be misled into checking rec.diseases rather than the mapped rec.disease.

greptile-apps · 2026-05-04T17:03:53Z

+    // the cache; the seeder's TTL + per-source lookback already bound
+    // freshness at write time. PR #3593: production saw 50 valid records
+    // cached but 0 rendered because the newest CDC item was 11d old.
+    const filteredDiseaseOutbreaks = mapLayers.diseaseOutbreaks ? this.diseaseOutbreaks : [];
    const filteredRadiationObservations = mapLayers.radiationWatch ? this.filterByTimeCached(this.radiationObservations, (obs) => obs.observedAt) : [];
    const filteredPositiveEvents = mapLayers.positiveEvents ? this.filterByTimeCached(this.positiveEvents, (e) => e.timestamp) : [];
    const filteredIranEvents = mapLayers.iranAttacks ? this.filterByTimeCached(this.iranEvents, (e) => e.timestamp) : [];


Disease outbreaks layer silently ignores the global time-range selector

The layer now bypasses filterByTimeCached entirely, so a user who sets the dropdown to "1h", "24h", or any other window will still see all cached items — including TGH alerts up to 90 days old. Every other layer (earthquakes, natural events, radiation, etc.) honors the selector; disease outbreaks is the only exception. There's no UI affordance (e.g. a layer-level tooltip or disabled-filter indicator) to tell users the time filter doesn't apply here, which may cause silent confusion.

The seeder-side freshness bound is a reasonable architectural argument, but the asymmetry with every other map layer is worth surfacing to users rather than silently diverging from the expected time-filter contract.

Captures the architectural follow-up identified during the 2026-05-04 disease-outbreaks incident: /api/health currently reports seeder-run freshness, not content freshness. For sparse upstream sources (WHO Disease Outbreak News, IEA OPEC reports, central-bank releases, WB annual indicators) these diverge — seeder runs fine, seed-meta fetchedAt stays fresh, but the freshest item the user sees is days or weeks old. Health says OK; UI renders nothing. Plan opts seeders into a parallel content-age contract: - runSeed accepts itemTimestamp / itemsPath / maxContentAgeMin - seed-meta carries newestItemAt/oldestItemAt/maxContentAgeMin when set - api/health reports new STALE_CONTENT status when content is older than the seeder's content-age budget Backwards compatible — legacy seeders without itemTimestamp keep current behavior. Pilot on disease-outbreaks (today's incident's origin), then migrate sparse + annual seeders over Sprints 3-4. Companion to PRs #3582 (canonical-envelope-mirror), #3593 (disease- outbreaks TGH + time-filter fixes), and the broader 'fetched-recently is not the same as fresh-content' insight.

…revisions (#3595) Five rounds of Codex review against plan #3594: - Round 1 (8 findings): pilot threshold won't catch incident; canonical-mirror loses content fields; synthetic timestamps mask staleness; all-undated falls through to OK; wrong health.js target symbols; soft-disabled budget; brittle autodetect; missing tests. Adopted contentMeta(data) -> {newestItemAt, oldestItemAt} API. - Round 2 (3 findings): envelope-writer chain incomplete (need _seed-envelope-source.mjs + parity mirrors + _seed-contract.mjs); classifier precedence wrong; disease snippet broke isNaN filters and mapItem. - Round 3 (3 findings): TGH source missed migration; stale classifier code block contradicted Sprint 1; helpers leaked via list-disease-outbreaks + bootstrap. - Round 4 (3 findings): replacement classifier still used bare 'return' but real code uses status='X' assignment; mapItem section contradicted strip contract; grep missed _originalPublishedMs. - Round 5 (1 finding): test descriptions still said cached items 'have helper fields'; rewrote to separate pre-publish in-memory layer from post-strip published-canonical layer. Net: ~280 prod LOC + ~250 test LOC for Sprint 1; explicit envelope-writer chain coverage; Sprint 2 disease pilot covers all 3 sources (WHO/RSS/TGH) + helper-field strip via publishTransform + anti-regression tests. Companion: PR #3593 (immediate disease-outbreaks fixes), #3582 (canonical-envelope-mirror).

…kip time filter (koala73#3593) PRODUCTION 2026-05-04: enabling the disease-outbreaks map layer renders nothing despite /api/health reporting `diseaseOutbreaks: status=OK, records=50`. Direct Redis read confirmed the cache holds 50 items but none render. Three compound issues, fixed in one PR. == A. ThinkGlobalHealth source returning 0 items == Two sub-bugs in `fetchThinkGlobalHealth`: 1. Wrong default branch in URL. The seeder hardcoded `/main/index_bundle.js` but the TGH GitHub repo's default branch is `master`. The `/main/` URL has been returning HTTP 404 silently — the seeder caught the !resp.ok and returned []. Fix: `main` → `master`. Verified live: 200 OK, 7.5 MB bundle. 2. Bundle format change after 2026-04 webpack rebuild. The legacy parser anchored on `var a=[{Alert_ID:` (unquoted JS keys). The new bundle wraps records in `eval("var res = [...]")` blocks with JSON-quoted keys like `\"Alert_ID\":\"8732529\"`. The old regex (`/(\w+):"((?:[^"\\]|\\.)*)"/`) doesn't match the quoted-key form. Fix: import `parseRealtimeAlerts` from seed-vpd-tracker.mjs (which already handles this exact format with a battle-tested schema-anchored scanner — VPD and Disease share the same TGH bundle). After both fixes, TGH contributes ~1,600 ProMED-reviewed alerts with real lat/lng, restoring the only geo-rich source (WHO/CDC/ONT only publish country names → require centroid fallback). == B. UI time filter eats every item == The map's `filterByTimeCached` (DeckGLMap.ts:1505) gated diseaseOutbreaks by the global timeRange dropdown (max '7d'). Disease outbreaks are sparse-by-nature — WHO DON publishes 1-2/week, CDC HAN alerts are infrequent, TGH carries 90 days of ProMED items. When the most recent WHO/CDC update is 8+ days old (normal), the 7d gate dropped every item → empty layer. Production confirmed: 50 cached items, newest 11.0 days old, all dropped. Fix: skip the time filter for diseaseOutbreaks. Render all items in the cache; the seeder's per-source lookback already bounds freshness at write time. Other layers keep the global filter. == Out of scope == - C: structural health-readiness probe (seedAgeMin tracks seeder run, not item freshness — separate followup). - Static-layer zoom-gates (bases/nuclear/spaceports/economic show nothing at default zoom 2 because LAYER_ZOOM_THRESHOLDS[*].minZoom is 3-5). Intentional UX, not a data bug — separate followup if we want a "zoom in to see N items" affordance.

…a73#3594) Captures the architectural follow-up identified during the 2026-05-04 disease-outbreaks incident: /api/health currently reports seeder-run freshness, not content freshness. For sparse upstream sources (WHO Disease Outbreak News, IEA OPEC reports, central-bank releases, WB annual indicators) these diverge — seeder runs fine, seed-meta fetchedAt stays fresh, but the freshest item the user sees is days or weeks old. Health says OK; UI renders nothing. Plan opts seeders into a parallel content-age contract: - runSeed accepts itemTimestamp / itemsPath / maxContentAgeMin - seed-meta carries newestItemAt/oldestItemAt/maxContentAgeMin when set - api/health reports new STALE_CONTENT status when content is older than the seeder's content-age budget Backwards compatible — legacy seeders without itemTimestamp keep current behavior. Pilot on disease-outbreaks (today's incident's origin), then migrate sparse + annual seeders over Sprints 3-4. Companion to PRs koala73#3582 (canonical-envelope-mirror), koala73#3593 (disease- outbreaks TGH + time-filter fixes), and the broader 'fetched-recently is not the same as fresh-content' insight.

…revisions (koala73#3595) Five rounds of Codex review against plan koala73#3594: - Round 1 (8 findings): pilot threshold won't catch incident; canonical-mirror loses content fields; synthetic timestamps mask staleness; all-undated falls through to OK; wrong health.js target symbols; soft-disabled budget; brittle autodetect; missing tests. Adopted contentMeta(data) -> {newestItemAt, oldestItemAt} API. - Round 2 (3 findings): envelope-writer chain incomplete (need _seed-envelope-source.mjs + parity mirrors + _seed-contract.mjs); classifier precedence wrong; disease snippet broke isNaN filters and mapItem. - Round 3 (3 findings): TGH source missed migration; stale classifier code block contradicted Sprint 1; helpers leaked via list-disease-outbreaks + bootstrap. - Round 4 (3 findings): replacement classifier still used bare 'return' but real code uses status='X' assignment; mapItem section contradicted strip contract; grep missed _originalPublishedMs. - Round 5 (1 finding): test descriptions still said cached items 'have helper fields'; rewrote to separate pre-publish in-memory layer from post-strip published-canonical layer. Net: ~280 prod LOC + ~250 test LOC for Sprint 1; explicit envelope-writer chain coverage; Sprint 2 disease pilot covers all 3 sources (WHO/RSS/TGH) + helper-field strip via publishTransform + anti-regression tests. Companion: PR koala73#3593 (immediate disease-outbreaks fixes), koala73#3582 (canonical-envelope-mirror).

vercel Bot deployed to Preview May 4, 2026 17:02 View deployment

greptile-apps Bot reviewed May 4, 2026

View reviewed changes

koala73 merged commit bd9c596 into main May 4, 2026
12 checks passed

koala73 deleted the fix/disease-outbreaks-tgh-404-and-time-filter branch May 4, 2026 17:09

koala73 mentioned this pull request May 4, 2026

feat(disease-outbreaks): Sprint 2 — content-age pilot (synthetic tagging + 9d budget) #3597

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(disease-outbreaks): TGH master URL + new bundle parser + skip time filter#3593

fix(disease-outbreaks): TGH master URL + new bundle parser + skip time filter#3593
koala73 merged 1 commit into
mainfrom
fix/disease-outbreaks-tgh-404-and-time-filter

koala73 commented May 4, 2026

Uh oh!

vercel Bot commented May 4, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented May 4, 2026

Uh oh!

greptile-apps Bot May 4, 2026

Uh oh!

greptile-apps Bot May 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented May 4, 2026

A. ThinkGlobalHealth source returning 0 items

B. UI time filter eats every item

Out of scope

Test plan

Uh oh!

vercel Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented May 4, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot May 4, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented May 4, 2026 •

edited

Loading