Skip to content

feat(energy): Energy Atlas end-to-end — pipelines + storage + shortages + disruptions + country drill-down#3294

Merged
koala73 merged 18 commits into
mainfrom
feat/energy-atlas-pipelines
Apr 23, 2026
Merged

feat(energy): Energy Atlas end-to-end — pipelines + storage + shortages + disruptions + country drill-down#3294
koala73 merged 18 commits into
mainfrom
feat/energy-atlas-pipelines

Conversation

@koala73

@koala73 koala73 commented Apr 22, 2026

Copy link
Copy Markdown
Owner

Summary

Full Energy Atlas feature for the energy.worldmonitor.app variant, implementing Release 1 scope of the Global Energy Flow parity plan (4-week plan, internal doc). 17 commits, 69 files, +9,663 lines. Ready for merge.

Five registries + nine RPCs + three panels + three DeckGL map layers + drill-down + methodology, all behind the existing energy variant scaffold shipped in #3291.

What's in here

Registries (curated v1, classifier-extended post-launch)

Registry Key Entries Cadence
Pipelines — gas energy:pipelines:gas:v1 12 weekly
Pipelines — oil energy:pipelines:oil:v1 12 weekly
Storage facilities energy:storage-facilities:v1 21 (UGS/SPR/LNG/crude hubs) weekly
Fuel shortages energy:fuel-shortages:v1 15 (confirmed + watch) daily
Energy disruptions energy:disruptions:v1 12 event log weekly

Every entry carries a versioned evidence bundle (physicalState, commercialState, sanctionRefs[], operatorStatement, classifierVersion, classifierConfidence, lastEvidenceUpdate). Public badges (flowing/reduced/offline/disputed for assets, confirmed/watch for shortages) are either derived server-side from evidence or emitted directly by the classifier — never hand-labeled.

All five registries registered across the 4-file bootstrap checklist (cache-keys.ts, api/bootstrap.js, api/health.js, api/seed-health.js) with correct maxStaleMin and intervalMin mirroring cron cadence.

RPCs (all under /api/supply-chain/v1, gateway cache-tier set)

  • ListPipelines + GetPipelineDetail
  • ListStorageFacilities + GetStorageFacilityDetail
  • ListFuelShortages + GetFuelShortageDetail
  • ListEnergyDisruptions

Every handler projects raw Redis → wire format through a pure projection function that attaches the derived public_badge. upstreamUnavailable is correctly gated on raw-fetch-null, NOT on post-filter row count (fixed in 20341c324 after P2 review — see upstream-unavailable-vs-empty-filter skill).

Shared deriver modules (src/shared/)

Duck-typed, dependency-free so the same pure function powers both the RPC handler projection AND the panel's bootstrap-first-paint projection. Identical output in both paths → no hydration flicker.

  • pipeline-evidence.tsderivePipelinePublicBadge + classifier-version picker + ISO timestamp picker
  • storage-evidence.tsderiveStoragePublicBadge
  • shortage-evidence.tsderiveShortageEvidenceQuality (trust hint only, NOT severity — that's classifier output)
  • disruption-timeline.ts — formatters for the shared timeline UI
  • pipeline-registry-store.ts / storage-facility-registry-store.ts / fuel-shortage-registry-store.ts — drain-once memo stores so panel + DeckGL layer don't race on the single-use getHydratedData cache

Panels (energy variant only)

  • PipelineStatusPanel — table + evidence drawer + disruption timeline
  • StorageFacilityMapPanel — table + evidence drawer + disruption timeline
  • FuelShortagePanel — severity-sorted table + source-typed evidence list
  • Clickable map ↔ panel coupling via window.dispatchEvent('energy:open-*-detail') (loose coupling — no-ops when the target panel isn't mounted)

DeckGL map layers (energy variant only)

  • Pipelines PathLayer — colored by derived badge, highlight support, updateTriggers on signature
  • Storage ScatterplotLayer — log-scale radius by capacity, color by badge
  • Fuel shortage pins — country-centroid placement with angular offset when multiple shortages share a country; severity coloring

Country drill-down

CountryDeepDivePanel.renderEnergyProfile gains an "Atlas exposure" sub-section per country showing:

  • Pipelines touching this country (from/to/transit filter)
  • Storage facilities in this country
  • Active fuel shortages in this country (severity breakdown)

Rows dispatch the same panel-open events — click-through works on the energy variant, silently no-ops elsewhere. Dynamic imports keep the three stores out of non-energy variant bundles.

Methodology pages

Public docs at /docs/methodology/{pipelines,storage,shortages,chokepoints,disruptions}. Each explains scope, evidence schema, what the classifier does vs. what humans do, public API, versioning, and the revision-log audit path.

Tests

  • 9 new registry/store/evidence test files (~80 new test cases)
  • Full unit suite: 6,426 tests, all passing
  • Typecheck + typecheck:api clean
  • Pre-push guards (full test suite, lint, boundaries, proto freshness, version sync, edge-function bundling, Unicode safety) pass on every commit

Commit log (stacked)

```
20341c3 fix(energy): upstreamUnavailable only fires when Redis returned nothing
3ff0151 docs(energy): methodology page for energy disruption event log
6999138 feat(energy): Day 14 — country drill-down Atlas exposure section
ccce482 feat(energy): Day 13 — energy disruption event log + asset timeline drawer
44e6d78 feat(energy): Day 12 — FuelShortagePanel + DeckGL shortage pins
0454a9b feat(energy): Day 11 — fuel-shortage registry schema + seed + RPC
d78bab4 feat(energy): Day 10 — storage atlas (ListStorageFacilities RPC + DeckGL layer + panel)
c75a2ac feat(energy): Day 9 — storage facility registry (UGS + SPR + LNG + crude hubs)
57f4341 fix(energy): three PR-review findings on Day 8b DeckGL integration
2e70d05 feat(energy): DeckGL PathLayer colored by evidence-derived badge
db52965 fix(energy): three PR-review P2s on PipelineStatusPanel + aggregators
149d33e fix(energy): PipelineStatusPanel bootstrap path — client-side badge derivation
792eb1a feat(energy): PipelineStatusPanel — evidence-backed status table + drawer
0f60611 feat(energy): ListPipelines + GetPipelineDetail RPCs with evidence-derived badges
6b01fa5 fix(energy): complete pipeline bootstrap registration per 4-file checklist
fd0adcd fix(energy): split seed-pipelines.mjs into two entry points
c49ab39 feat(energy): pipeline registries (gas + oil) — evidence-based schema
```

Design principles enforced

  • We ship evidence, not conclusions. Public badges are deterministic functions of evidence bundles, versioned so consumers can pin a reader to a classifier version.
  • Bootstrap first-paint must match post-RPC render. Shared deriver pattern prevents the class of bugs where raw JSON crashes on a server-computed field (see bootstrap-hydration-derived-field-parity skill).
  • Single-use bootstrap cache cannot race. Shared drain-once stores let map + panel read identical data from one drain (see same skill).
  • runSeed() hard-exits. Two runSeed calls in one process = second key silently unwritten. Pipelines split into two entries accordingly (see runseed-then-after-process-exit skill).
  • Agent-native from day one. Every registry has proto + OpenAPI + gateway cache-tier, so MCP tools / external agents consume the same data as the UI, derived by the same versioned classifier.

Known follow-ups (intentionally deferred to post-launch per plan)

  • Classifier implementation in `proactive-intelligence.mjs` — v1 ships with curated seeds; classifier extends the same Redis keys automatically post-launch.
  • `/corrections` auto-revision-log page — schema is ready (classifier writes entries as a byproduct of normal operation); public page is a small Week 4+ task.
  • `getEnergyShock` cascade RPC, scenario runner, global tanker density, shadow-fleet layer — all Release 2 (monetization phase) per plan.

Test plan

  • `npm run typecheck` (both configs) clean on every commit
  • `npx tsx --test tests/` all 6,426 unit tests pass
  • Pre-push full-suite + lint + proto freshness + version sync + Unicode safety pass on every push
  • Reviewer caught P2 on `upstreamUnavailable` semantics — fixed in `20341c324`, sibling handlers audited
  • Smoke test `energy.worldmonitor.app` bootstrap after deploy (panel drawers, map layer clicks, country drill-down)
  • Verify Railway cron services provisioned for the 5 seeders before treating "no data yet" as healthy
  • Confirm CF Worker CORS rules allow the new `/api/supply-chain/v1/list-*` paths (inherited from existing supply-chain base)

Related

@vercel

vercel Bot commented Apr 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment Apr 23, 2026 3:27am

Request Review

Base automatically changed from feat/energy-atlas-release-1 to main April 22, 2026 17:37
koala73 added 3 commits April 22, 2026 21:38
Day 6 of the Energy Atlas Release 1 plan (Week 2). First curated asset
registry for the atlas — the real gap vs GEF.

## Curated data (critical assets only, not global completeness)

scripts/data/pipelines-gas.json — 12 critical gas lines:
  Nord Stream 1/2 (offline; Swedish EEZ sabotage 2022; EU sanctions refs),
  TurkStream, Yamal–Europe (offline; Polish counter-sanctions),
  Brotherhood/Soyuz (offline; Ukraine transit expired 2024-12-31),
  Power of Siberia, Dolphin, Medgaz, TAP, TANAP,
  Central Asia–China, Langeled.

scripts/data/pipelines-oil.json — 12 critical oil lines:
  Druzhba North/South (N offline per EU 2022/879; S under landlocked
  derogation), CPC, ESPO (+ price-cap sanction ref), BTC, TAPS,
  Habshan–Fujairah (Hormuz bypass), Keystone, Kirkuk–Ceyhan (offline
  since 2023 ICC ruling), Baku–Supsa, Trans-Mountain (TMX expansion
  May 2024), ESPO spur to Daqing.

Scope note: 75+ each is Week 2b work via GEM bulk import. Today's cut
is curated from first-hand operator disclosures + regulator filings so
I can stand behind every evidence field.

## Evidence-based schema (not conclusion labels)

Per docs/methodology/pipelines.mdx: no bare `sanctions_blocked` field.
Every pipeline carries an evidence bundle with `physicalState`,
`physicalStateSource`, `operatorStatement`, `commercialState`,
`sanctionRefs[]`, `lastEvidenceUpdate`, `classifierVersion`,
`classifierConfidence`. The public badge (`flowing|reduced|offline|
disputed`) is derived server-side from this bundle at read time.

## Seeder

scripts/seed-pipelines.mjs — single process publishes BOTH keys
(energy:pipelines:{gas,oil}:v1) via two runSeed() calls. Tiny datasets
(<20KB each) so co-location is cheap and guarantees classifierVersion
consistency.

Conventions followed (worldmonitor-bootstrap-registration skill):
- TTL 21d = 3× weekly cadence (gold-standard per
  feedback_seeder_gold_standard.md)
- maxStaleMin 20_160 = 2× cadence (health-maxstalemin-write-cadence skill)
- sourceVersion + schemaVersion + recordCount + declareRecords wired
  (seed-contract-foundation)
- Zero-case explicitly NOT allowed — MIN_PIPELINES_PER_REGISTRY=8 floor

## Health registration (dual, per feedback_two_health_endpoints_must_match)

- api/health.js: BOOTSTRAP_KEYS adds pipelinesGas + pipelinesOil;
  SEED_META adds both with maxStaleMin=20_160.
- api/seed-health.js: mirror entries with intervalMin=10_080 (maxStaleMin/2).

## Bundle registration

scripts/seed-bundle-energy-sources.mjs adds a single Pipelines entry
(not two) because seed-pipelines.mjs publishes both keys in one run —
listing oil separately would double-execute. Monitoring of the oil key
staleness happens in api/health.js instead.

## Tests (tests/pipelines-registry.test.mts)

17 passing node:test assertions covering:
- Schema validation (both registries pass validateRegistry)
- Identity resolution (no id collisions, id matches object key)
- Country ISO2 normalization (from/to/transit all match /^[A-Z]{2}$/)
- Endpoint geometry within Earth bounds
- Evidence rigor: non-flowing badges require at least one supporting
  evidence source (operator statement / sanctionRefs / ais-relay /
  satellite / press)
- ClassifierConfidence in 0..1
- Commodity/capacity pairing (gas uses capacityBcmYr, oil uses
  capacityMbd — mixing = test fail)
- validateRegistry rejects: empty object, null, no-evidence fixtures,
  below-floor counts

Typecheck clean (both tsconfig.json and tsconfig.api.json).

Next: Day 7 will add list-pipelines / get-pipeline-detail RPCs in
supply-chain/v1. Day 8 ships PipelineStatusPanel with DeckGL PathLayer
consuming the registry.
… hard-exits

High finding from PR review. scripts/seed-pipelines.mjs called runSeed()
twice in one process and awaited Promise.all. But runSeed() in
scripts/_seed-utils.mjs hard-exits via process.exit on ~9 terminal paths
(lines 816, 820, 839, 888, 917, 989, plus fetch-retry 946, fatal 859,
skipped-lock 81). The first runSeed to reach any terminal path exits the
entire node process, so the second runSeed's resolve never fires — only
one of energy:pipelines:{gas,oil}:v1 would ever be written.

Since the bundle scheduled seed-pipelines.mjs exactly once, and both
api/health.js and api/seed-health.js expect both keys populated, the
other registry would stay permanently EMPTY/STALE after deploy.

Fix: split into two entry-point scripts around a shared utility.

- scripts/_pipeline-registry.mjs (NEW, was seed-pipelines.mjs) — shared
  helpers ONLY. Exports GAS_CANONICAL_KEY, OIL_CANONICAL_KEY,
  PIPELINES_TTL_SECONDS, MAX_STALE_MIN, buildGasPayload, buildOilPayload,
  validateRegistry, recordCount, declareRecords. Underscore prefix marks
  it as non-entry-point (matches _seed-utils.mjs / _seed-envelope-source.mjs
  convention).
- scripts/seed-pipelines-gas.mjs (NEW) — imports from the shared module,
  single runSeed('energy','pipelines-gas',…) call.
- scripts/seed-pipelines-oil.mjs (NEW) — same shape, oil.
- scripts/seed-bundle-energy-sources.mjs — register BOTH seeders (not one).
- scripts/seed-pipelines.mjs — deleted.
- tests/pipelines-registry.test.mts — update import path to the shared
  module. All 17 tests still pass.

Typecheck clean (both configs). Tests pass. No other consumers import
from the deleted script.
…klist

High finding from PR review. My earlier PR description claimed
worldmonitor-bootstrap-registration was complete, but I only touched two
of the four registries (api/health.js + api/seed-health.js). The bootstrap
hydration payload itself (api/bootstrap.js) and the shared cache-keys
registry (server/_shared/cache-keys.ts) still had no entry for either
pipeline key, so any consumer that reads bootstrap data would see
pipelinesGas/pipelinesOil as missing on first load.

Files updated this commit:

- api/bootstrap.js — KEYS map + SLOW_KEYS set both gain pipelinesGas +
  pipelinesOil. Placed next to sprPolicies (same curated-registry cadence
  and tier). Slow tier is correct: weekly cron, not needed on first paint.
- server/_shared/cache-keys.ts — PIPELINES_GAS_KEY + PIPELINES_OIL_KEY
  exported constants (matches SPR_POLICIES_KEY pattern), BOOTSTRAP_KEYS map
  entries, and BOOTSTRAP_TIERS entries (both 'slow').

Not touched (intentional):
- server/gateway.ts — pipeline data is free-tier per the Energy Atlas
  plan; no PREMIUM_RPC_PATHS entry required. Energy Atlas monetization
  hooks (scenario runner, MCP tools, subscriptions) are Release 2.

Full 4-file checklist now complete:
  ✅ server/_shared/cache-keys.ts (this commit)
  ✅ api/bootstrap.js          (this commit)
  ✅ api/health.js             (earlier in PR)
  ✅ api/seed-health.js        (earlier in PR — dual-registry rule)

Typecheck clean (both configs).
…rived badges

Day 7 of the Energy Atlas Release 1 plan (Week 2). Exposes the pipeline
registries (shipped in Day 6) via two supply-chain RPCs and ships the
evidence-to-badge derivation server-side.

## Proto

proto/worldmonitor/supply_chain/v1/list_pipelines.proto — new:
- ListPipelinesRequest { commodity_type?: 'gas' | 'oil' }
- ListPipelinesResponse { pipelines[], fetched_at, classifier_version, upstream_unavailable }
- GetPipelineDetailRequest { pipeline_id (required, query-param) }
- GetPipelineDetailResponse { pipeline?, revisions[], fetched_at, unavailable }
- PipelineEntry — wire shape mirroring scripts/data/pipelines-{gas,oil}.json
  + a server-derived public_badge field
- PipelineEvidence, OperatorStatement, SanctionRef, LatLon, PipelineRevisionEntry

service.proto adds both rpc methods with HTTP_METHOD_GET + path bindings:
  /api/supply-chain/v1/list-pipelines
  /api/supply-chain/v1/get-pipeline-detail

`make generate` regenerated src/generated/{client,server}/… + docs/api/
OpenAPI json/yaml.

## Evidence-derivation

server/worldmonitor/supply-chain/v1/_pipeline-evidence.ts — new.
derivePublicBadge(evidence) → 'flowing' | 'reduced' | 'offline' | 'disputed'
is deterministic + versioned (DERIVER_VERSION='badge-deriver-v1').

Rules (first match wins):
1. offline + sanctionRef OR expired/suspended commercial → offline
2. offline + operator statement → offline
3. offline + only press/ais/satellite → disputed (single-source negative claim)
4. reduced → reduced
5. flowing → flowing
6. unknown / malformed → disputed

Staleness guard: non-flowing badges on >14d-old evidence demote to
disputed. Flowing is the optimistic default — stale "still flowing" is
safer than stale "offline". Matches seed-pipelines-{gas,oil}.mjs maxStaleMin.

Tests (tests/pipeline-evidence-derivation.test.mts) — 15 passing cases
covering happy paths, disputed fallbacks, staleness guard, versioning.

## Handlers

server/worldmonitor/supply-chain/v1/list-pipelines.ts
- Reads energy:pipelines:{gas,oil}:v1 via getCachedJson.
- projectPipeline() narrows the Upstash `unknown` into PipelineEntry
  shape + calls derivePublicBadge.
- Honors commodity_type filter (skip the opposite registry's Redis read
  when the client pre-filters).
- Returns upstream_unavailable=true when BOTH registries miss.

server/worldmonitor/supply-chain/v1/get-pipeline-detail.ts
- Scans both registries by id (ids are globally unique per
  tests/pipelines-registry.test.mts).
- Empty revisions[] for now; auto-revision log wires up in Week 3.

handler.ts registers both into supplyChainHandler.

## Gateway

server/gateway.ts adds 'static' cache-tier for both new RPC paths
(registry is slow-moving; 'static' matches the other read-mostly
supply-chain endpoints).

## Consumer wiring

Not in this commit — PipelineStatusPanel (Day 8) is what will call
listPipelines/getPipelineDetail via the generated client. pipelinesGas
+ pipelinesOil stay in PENDING_CONSUMERS until Day 8.

Typecheck clean (both configs). 15 new tests + 17 registry tests all pass.
…awer

Day 8 of the Energy Atlas Release 1 plan. First consumer of the Day 6–7
registries + RPCs.

## What this PR adds

- src/components/PipelineStatusPanel.ts — new panel (id=pipeline-status).
  * Bootstrap-hydrates from pipelinesGas + pipelinesOil for instant first
    paint; falls through to listPipelines() RPC if bootstrap misses.
    Background re-fetch runs on every render so a classifier-version bump
    between bootstrap stamp and first view produces a visible update.
  * Table rows sorted non-flowing-first (offline / reduced / disputed
    before flowing) — what an atlas reader cares about.
  * Click-to-expand drawer calls getPipelineDetail() lazily — operator
    statements, sanction refs (with clickable source URLs), commercial
    state, classifier version + confidence %, capacity + route metadata.
  * publicBadge color-chip palette matches the methodology doc.
  * Attribution footer with GEM (CC-BY 4.0) credit + classifier version.

- src/components/index.ts — barrel export.
- src/app/panel-layout.ts — import + createPanel('pipeline-status', …).
- src/config/panels.ts — ENERGY_PANELS adds 'pipeline-status' at priority 1.

## PENDING_CONSUMERS cleanup

tests/bootstrap.test.mjs — removes 'pipelinesGas' + 'pipelinesOil' from
the allowlist. The invariant "every bootstrap key has a getHydratedData
consumer" now enforces real wiring for these keys: the panel literally
calls getHydratedData('pipelinesGas') and getHydratedData('pipelinesOil').
Future regressions that remove the consumer will fail pre-push.

## Consumer contract verified

- 67 tests pass including bootstrap.test.mjs consumer coverage check.
- Typecheck clean.
- No DeckGL PathLayer in this commit — existing 'pipelines-layer' has a
  separate data source, so modifying DeckGLMap.ts to overlay evidence-
  derived badges on the map is a follow-up commit to avoid clobbering.

## Out of scope for Day 8 (next steps on same PR)

- DeckGL PathLayer integration (color pipelines on the main map by
  publicBadge, click-to-open this drawer) — Day 8b commit.
- Storage facility registry + StorageFacilityMapPanel — Days 9-10.
…erivation

High finding from PR review. The Day-8 panel crashed on first paint
whenever bootstrap hydration succeeded, because:

- Bootstrap hydrates raw scripts/data/pipelines-{gas,oil}.json verbatim.
- That JSON does NOT include publicBadge — that field is only added by
  the server handler's projectPipeline() in list-pipelines.ts.
- PipelineStatusPanel passed raw entries into badgeChip(), which called
  badgeLabel(undefined).charAt(0) → TypeError.

The background RPC refresh that would have repaired the data never ran
because the panel threw before reaching it. So the exact bootstrap path
newly wired in commit 6b01fa5 was broken for the new panel.

Fix: move the evidence→badge deriver to src/shared/pipeline-evidence.ts
so the client panel and the server handler run the identical function on
identical inputs. Panel projects raw bootstrap JSON through the shared
deriver client-side, producing the same publicBadge the RPC would have
returned. No UI flicker on hydration because pre- and post-RPC badges
match exactly (same function, same input).

## Changes

- src/shared/pipeline-evidence.ts (NEW) — pure deriver with duck-typed
  PipelineEvidenceInput (no generated-type dependency, so both client
  and server assign their proto-typed evidence bundles by structural
  subtyping). Exports derivePipelinePublicBadge + version + type.
- server/worldmonitor/supply-chain/v1/_pipeline-evidence.ts — now a thin
  re-export of the shared module under its older name so in-handler
  imports keep working without a sweep.
- src/components/PipelineStatusPanel.ts:
  * Imports derivePipelinePublicBadge from @/shared/pipeline-evidence.
  * NEW projectRawPipeline() defensively coerces every field from
    unknown → PipelineEntry shape, mirroring the server projection.
  * buildBootstrapResponse now routes every raw entry through the
    projection before returning, so the wire-format PipelineEntry[] the
    renderer receives always has publicBadge populated.
  * badgeChip() gained a null-guard fallback to 'disputed' — belt +
    braces so even if a future caller passes an undefined, the UI
    renders safely instead of throwing.
  * BootstrapRegistry renamed RawBootstrapRegistry with a comment
    explaining why the seeder ships raw JSON (not wire format).

## Regression tests

tests/pipeline-panel-bootstrap.test.mts (NEW) — 6 tests that exercise
the bootstrap-first-paint path end-to-end:

- Every gas + oil curated entry produces a valid badge.
- Raw entries never ship with pre-computed publicBadge (contract guard
  on the seed data format).
- Deriver never throws on undefined/null/{} evidence (was the crash).
- Nord Stream 1 regression check (offline + paperwork → offline).
- Druzhba-South staleness behavior (reduced when fresh, disputed after
  60 days without update).

38/38 tests now pass (17 registry + 15 deriver + 6 new bootstrap-path).
Typecheck clean on both configs.

## Invariant preserved

The server handler and the panel render identical badges because:
1. Same pure function (imported from the same module).
2. Same deterministic rules, same staleness window.
3. Same bootstrap data read by both paths (Redis → either bootstrap
   payload or RPC response).

No UI flicker on hydration.
## P2-1 — sanitizeUrl on external evidence links (XSS hardening)

Sanction-ref URLs and operator-statement URLs were interpolated with
escapeHtml only. HTML-escaping blocks tag injection but NOT javascript:
or data: URL schemes, so a bad URL in the seeded registry would execute
in-app when a reader clicked the evidence link. Every other panel in
the codebase (NewsPanel, GdeltIntelPanel, GeoHubsPanel, AirlineIntelPanel,
MonitorPanel) uses sanitizeUrl for this exact reason.

Fix: import sanitizeUrl from @/utils/sanitize and route both hrefs
through it. sanitizeUrl() drops non-http(s) schemes + returns '' on
invalid URLs. The renderer now suppresses the <a> entirely when
sanitize rejects — the date label still renders as plain text instead
of becoming an executable link.

## P2-2 — loadDetail catch path missing stale-response guard

The success path at loadDetail() checked `this.selectedId !== pipelineId`
to suppress stale responses when the user has clicked another pipeline
mid-flight. The catch path at line 219 had no such guard: if the user
clicked A, then B, and A's request failed before B resolved, A's error
handler cleared detailLoading and detail, showing "Pipeline detail
unavailable" for B's drawer even though B was still loading.

Fix: mirror the same `if (this.selectedId !== pipelineId) return` guard
in the catch path. The newer request now owns the drawer state
regardless of which path (success OR failure) the older one took.

## P2-3 — always-gas-preference aggregator for classifierVersion + fetchedAt

Three call sites (list-pipelines.ts handler, get-pipeline-detail.ts
handler, PipelineStatusPanel bootstrap projection) computed aggregate
classifier version and fetchedAt by `gas?.x || oil?.x || fallback`.
That was defensible when a single seed-pipelines.mjs wrote both keys
atomically (fix commit 29b4ac7 split this into two separate Railway
cron entry points). Now gas + oil cron independently, so mixed-version
(gas=v1, oil=v2 during classifier rollout) and mixed-timestamp (oil
refreshed 6h after gas) windows are the EXPECTED state, not the
exceptional one. The comment in list-pipelines.ts even said "pick the
newest classifier version" but the code didn't actually compare.

Fix: add two shared helpers in src/shared/pipeline-evidence.ts —

- pickNewerClassifierVersion(a,b) — parses /^v(\\d+)$/ and returns the
  higher-numbered version; falls back to lexicographic for non-v-
  prefixed values; handles single-missing inputs.
- pickNewerIsoTimestamp(a,b) — Date.parse()-compares and returns the
  later ISO; handles missing / malformed inputs gracefully.

Both server RPCs and the panel bootstrap projection now call these
helpers identically, so clients are told the truth about version +
freshness during partial rollouts.

## Tests

Extended tests/pipeline-evidence-derivation.test.mts with 8 new
assertions covering both pickers:

- Higher v-number wins regardless of order (v1 vs v2 → v2 both ways)
- Single-missing falls back to the one present
- Missing + missing → default 'v1' for version / '' for ts
- Non-v-numbered values fall back to lexicographic
- Explicit regression: "gas=v1 + oil=v2 during rollout" returns v2
- Explicit regression: "oil fresher than gas" returns the oil timestamp

38 → 46 tests. All pass. Typecheck clean on both configs.
…p↔panel link

Day 8b of the Energy Atlas plan. Pipelines now render on the main
DeckGL map of the energy variant colored by their derived publicBadge,
and clicking a pipeline on the map opens the same evidence drawer the
panel row-click opens.

## Why this commit

Day 8 shipped the PipelineStatusPanel as a table + drawer view.
Reviewer flag notwithstanding (fixed in 149d33e + db52965), a
table-only pipeline view is a weak product compared to the map-centric
atlas it's meant to rival. The map-layer differentiation is the whole
point of the feature.

## What this adds

src/components/DeckGLMap.ts:
- New createEnergyPipelinesLayer() — reads hydrated pipeline registries
  via getHydratedData, projects raw JSON through the shared deriver
  (src/shared/pipeline-evidence.ts), renders a DeckGL PathLayer colored
  by publicBadge:
    flowing  → green (46,204,113)
    reduced  → amber (243,156,18)
    offline  → red   (231,76,60)
    disputed → purple (155,89,182)
  Offline + disputed get thicker strokes (3px vs 2px) for at-a-glance
  surfacing of disrupted assets. Geometry comes from raw startPoint +
  waypoints[] + endPoint per asset (straight line when no waypoints).
- Branching at line ~1498: SITE_VARIANT === 'energy' routes to the
  new method; other variants keep the static PIPELINES config (colored
  by oil/gas type). Existing commodity/finance/full map layers are
  untouched — no cross-variant leakage.
- onClick handler emits `energy:open-pipeline-detail` as a window
  CustomEvent with { pipelineId }. Loose coupling: the map doesn't
  import the panel, the panel doesn't import the map.
- Fallback: if bootstrap hasn't hydrated yet, createEnergyPipelinesLayer
  falls back to the static createPipelinesLayer() so the pipelines
  toggle always shows *something*.

src/components/PipelineStatusPanel.ts:
- Constructor registers a window event listener for
  'energy:open-pipeline-detail' → calls this.loadDetail(pipelineId) →
  drawer opens on the clicked asset. Map click and row click converge
  on the same drawer, same evidence view.
- destroy() removes the listener to prevent ghost handlers after panel
  unmount.

## Guarantees

- Bootstrap parity: the DeckGL layer calls the SAME derivePipelinePublicBadge
  as the panel and the server handler, so the map color, the table row
  chip, and the RPC response all agree on the badge. No flicker, no
  drift, no confused user.
- Variant isolation: only SITE_VARIANT === 'energy' triggers the new
  path. Commodity / finance / full map layers untouched.
- No cross-component import: the panel doesn't reference the map class
  and vice versa. The event contract is the only coupling — testable,
  swappable, tauri-safe (guarded with `typeof window !== 'undefined'`).

Typecheck clean. PR #3294 now has 8 commits.

Follow-up backlog:
- Add waypoints[] to the curated pipelines-{gas,oil}.json so the map
  draws real routes instead of straight lines (cosmetic; does not
  affect correctness).
- Tooltip case in the picking tooltip registry (line ~3748) so hover
  shows "Nord Stream 1 · OFFLINE" before click.
## P1 — getHydratedData single-use race between map + panel

src/services/bootstrap.ts:34 — `if (val !== undefined) hydrationCache.delete(key);`
The helper drains its slot on first read. Day 8 (PipelineStatusPanel) and
Day 8b (createEnergyPipelinesLayer) BOTH call getHydratedData('pipelinesGas')
and getHydratedData('pipelinesOil') — whoever renders first drains the cache
and forces the loser onto its fallback path (panel → RPC, map → static
PIPELINES layer). The commit's "shared bootstrap-backed data" guarantee
did not actually hold.

Fix: new src/shared/pipeline-registry-store.ts that reads once and memoizes.
Both consumers read through getCachedPipelineRegistries() — same data, same
reference, unlimited re-reads. When the panel's background RPC fetch lands,
it calls setCachedPipelineRegistries() to back-propagate fresh data into
the store so the map's next re-render sees the newer classifierVersion +
fetchedAt too (no map/panel drift during classifier rollouts).

Test-only injection hook (__setBootstrapReaderForTests) makes the drain-once
semantics observable without a real bootstrap payload.

## P2 — pipelines-layer tooltip regresses to blank label on energy variant

src/components/DeckGLMap.ts:3748 (pipelines-layer tooltip case) still assumed
the static-config shape (obj.type). The new energy layer emits objects with
commodityType + badge fields, so the tooltip's type-ternary fell through to
the generic fallback — hover rendered " pipeline" (empty leading commodity)
instead of "Nord Stream 1 · OFFLINE".

Fix: differentiate by presence of obj.badge (only the energy layer sets it).
On the energy variant, tooltip now reads name + commodity + badge. Static-
config variants (commodity / finance / full) keep their existing format
unchanged.

## P2 — createEnergyPipelinesLayer dropped highlightedAssets behavior

The static createPipelinesLayer() reads this.highlightedAssets.pipeline and
threads it into getColor / getWidth with an updateTrigger on the signature.
Any caller using flashAssets('pipeline', [...]) or highlightAssets([...])
gets a visible red-outline flash on the matching paths. My Day 8b energy
layer ignored the set entirely — those APIs silently no-op'd on the energy
variant.

Fix: createEnergyPipelinesLayer() now reads the same highlight set, applies
HIGHLIGHT_COLOR + wider stroke to matching IDs, and wires
updateTriggers: { getColor: sig, getWidth: sig } so DeckGL actually
recomputes when the set changes.

Also removed the unnecessary layerCache.set() in the energy path: the
store can update via RPC back-propagation, and a cache keyed only on
highlight-signature would serve stale data. With ~25 critical-asset
pipelines, rebuild per render is trivial.

## Tests

tests/pipeline-registry-store.test.mts (NEW) — 5 tests covering the
drain-once read-many invariant: multiple consumers get cached data
without re-draining, RPC back-propagation updates the source, partial
updates preserve the other commodity, and pure RPC-first (no bootstrap)
works without invoking the reader.

All 51 PR tests pass. Typecheck clean on both configs.
…ude hubs)

Ships 21 critical strategic storage facilities as a curated registry, same
evidence-bundle pattern as the pipeline registries in Day 7/8:

- scripts/data/storage-facilities.json — 4 UGS + 4 SPR + 6 LNG export +
  3 LNG import + 4 crude tank farms. Each carries physicalState +
  sanctionRefs + classifierVersion/Confidence + fillDisclosed/fillSource.
- scripts/_storage-facility-registry.mjs — shared helpers (validator,
  builder, canonical key, MAX_STALE_MIN). Validator enforces facility-type
  × capacity-unit pairing (ugs→TWh, spr/tank-farm→Mb, LNG→Mtpa) and the
  non-operational badge ⇒ evidence invariant.
- scripts/seed-storage-facilities.mjs — single runSeed entry (only one
  key, so no split-seeder dance needed).
- Registered in the 4-file bootstrap checklist: cache-keys.ts
  (STORAGE_FACILITIES_KEY + BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS),
  api/bootstrap.js (KEYS + SLOW_KEYS), api/health.js (BOOTSTRAP_KEYS +
  SEED_META, 14d threshold = 2× weekly cron), api/seed-health.js (mirror).
- tests/bootstrap.test.mjs PENDING_CONSUMERS adds storageFacilities —
  Day 10 StorageFacilityMapPanel will remove it.
- tests/storage-facilities-registry.test.mts — 20 tests covering schema,
  identity, geometry, type×capacity pairing, evidence contract, and
  negative-input validator rejection.

Registry fields are slow-moving; badge derivation happens at read-time
server-side once the RPC handler lands in Day 10 (panel + deckGL
ScatterplotLayer). Seeded data is live in Redis from this commit so the
Day 10 PR only adds display surfaces.

Tests: 56 pass (36 prior + 20 new). Typecheck + typecheck:api clean.
…kGL ScatterplotLayer + panel)

End-to-end wiring for the strategic storage registry seeded in Day 9. Same
pattern as the pipeline shipping path (Days 7+8+8b): proto → handler →
shared evidence deriver → panel → DeckGL map layer, with a shared
read-once store keeping map + panel aligned.

Proto + generated code:
- list_storage_facilities.proto: ListStorageFacilities +
  GetStorageFacilityDetail messages with StorageFacilityEntry,
  StorageEvidence, StorageSanctionRef, StorageOperatorStatement,
  StorageLatLon, StorageFacilityRevisionEntry.
- service.proto wires both RPCs under /api/supply-chain/v1.
- make generate → regenerated client + server stubs + OpenAPI.

Server handlers:
- src/shared/storage-evidence.ts: shared pure deriver. Duck-typed input
  interface avoids generated-type deps; identical rules to the pipeline
  deriver (sanction/commercial paperwork vs external-signal-only offline,
  14d staleness window, version pin).
- _storage-evidence.ts: thin re-export for server handler import ergonomics.
- list-storage-facilities.ts: reads STORAGE_FACILITIES_KEY from Upstash,
  projects raw → wire format, attaches derived publicBadge, filters by
  optional facilityType query arg.
- get-storage-facility-detail.ts: single-asset lookup for drawer.
- handler.ts registers both new methods.
- gateway.ts: both routes → 'static' cache tier (registry is near-static).

Panel + map:
- src/shared/storage-facility-registry-store.ts: drain-once memo mirroring
  pipeline-registry-store. Both panel and DeckGL layer read through this
  so the single-use getHydratedData drain doesn't race between consumers.
  RPC back-propagation via setCachedStorageFacilityRegistry() keeps map ↔
  panel on the same classifierVersion during rollouts.
- StorageFacilityMapPanel.ts: table + evidence drawer. Bootstrap hot path
  projects raw registry through same deriver as server so first-paint
  badge matches post-RPC badge (no flicker). sanitizeUrl + stale-response
  guards (success + catch paths) carried over from PipelineStatusPanel.
- DeckGLMap.ts createEnergyStorageLayer(): ScatterplotLayer keyed on
  badge color; log-scale radius (6km–26km) keeps Rehden visible next to
  Ras Laffan. Click dispatches 'energy:open-storage-facility-detail' —
  panel listens and opens its drawer (loose coupling, no direct refs).
- Tooltip branch on storage-facilities-layer shows facility type, country,
  capacity unit, and badge.
- Added 'storageFacilities' optional field to MapLayers type (optional so
  existing variant literals across commodity/finance/tech/happy/full/etc.
  don't need touching). Wired into LAYER_REGISTRY + VARIANT_LAYER_ORDER.energy
  + ENERGY_MAP_LAYERS + ENERGY_MOBILE_MAP_LAYERS. Panel entry added to
  ENERGY_PANELS + panel-layout createPanel. PENDING_CONSUMERS entry from
  Day 9 removed — panel + map layer are now real consumers.

Tests:
- storage-evidence-derivation.test.mts (17 tests): covers every curated
  facility yields a valid badge, null/malformed input never throws,
  offline sanction/commercial/operator rules, external-signal-only offline
  → disputed, staleness demotion.
- storage-facility-registry-store.test.mts (4 tests): drain-once, no-data
  drain, RPC update, pure-RPC-first path.

All 6,426 unit tests pass. Typecheck + typecheck:api clean. Pre-existing
src-tauri/sidecar/ test failure is unrelated (no diff touches src-tauri/).
…assifier post-launch)

Ships v1 of the global fuel-shortage alert registry. Severity is the
CLASSIFIER OUTPUT (confirmed/watch), not a client derivation — we ship
the evidence alongside so readers can audit the grounds. v1 is seeded
from curated JSON; post-launch the proactive-intelligence classifier
(Day 12 work) extends the same key directly.

Data:
- scripts/data/fuel-shortages.json — 15 known active shortages
  (PK, LK, NG×2, CU, VE, LB, ZW, AR, IR, BO, KE, PA, EG, BY)
  spanning petrol/diesel/jet across confirmed + watch tiers. Each entry
  carries evidenceSources[] (regulator/operator/press), firstSeen,
  lastConfirmed, resolvedAt, impactTypes[], causeChain[], classifier
  version + confidence. Confirmed severity enforces authoritative
  evidence at schema level.

Seeder:
- scripts/_fuel-shortage-registry.mjs — shared validator (enforces
  iso2 country, enum products/severities/impacts/causes, authoritative
  evidence for confirmed). MIN_SHORTAGES=10.
- scripts/seed-fuel-shortages.mjs — single runSeed entry.
- Registered in seed-bundle-energy-sources.mjs at DAY cadence (shortages
  move faster than registry assets).

Bootstrap 4-file registration:
- cache-keys.ts: FUEL_SHORTAGES_KEY + BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS.
- api/bootstrap.js: KEYS + SLOW_KEYS.
- api/health.js: BOOTSTRAP_KEYS + SEED_META (2880min = 2× daily cron).
- api/seed-health.js: mirrors intervalMin=1440.

Proto + RPC:
- list_fuel_shortages.proto: ListFuelShortages (country/product/severity
  query facets) + GetFuelShortageDetail messages with FuelShortageEntry,
  FuelShortageEvidence, FuelShortageEvidenceSource.
- service.proto wires both new RPCs under /api/supply-chain/v1.
- list-fuel-shortages.ts handler projects raw → wire format, supports
  server-side country/product/severity filtering.
- get-fuel-shortage-detail.ts single-shortage lookup.
- handler.ts registers both. gateway.ts: 'medium' cache-tier (daily
  classifier updates warrant moderate freshness).

Shared evidence helper:
- src/shared/shortage-evidence.ts: deriveShortageEvidenceQuality maps
  (confidence + authoritative-source count + freshness) → 'strong' |
  'moderate' | 'thin' for client-side sort/trust indicators. Does NOT
  change severity — classifier owns that decision.
- countEvidenceSources buckets sources for the drawer's "n regulator /
  m press" line.

Tests:
- tests/fuel-shortages-registry.test.mts (19 tests): schema, identity,
  enum coverage, evidence contract (confirmed → authoritative source),
  validateRegistry negative cases.
- tests/shortage-evidence.test.mts (10 tests): quality deriver edge
  cases, source bucketing.
- tests/bootstrap.test.mjs PENDING_CONSUMERS adds fuelShortages —
  FuelShortagePanel arrives Day 12 which will remove the entry.

Typecheck + typecheck:api clean. 64 tests pass.
End-to-end wiring of the fuel-shortage registry shipped in Day 11: panel
on the Energy variant page, ScatterplotLayer pins on the DeckGL map,
both reading through a shared single-drain store so they don't race on
the bootstrap cache.

Panel:
- src/components/FuelShortagePanel.ts — table sorted by severity (confirmed
  first) then evidence quality (strong → thin) then most-recent lastConfirmed.
  Drawer shows short description, first-seen / last-confirmed / resolved,
  impact types, cause chain, classifier version/confidence, and a typed
  evidence-source list with regulator/operator/press chips. sanitizeUrl on
  every href so classifier-ingested URLs can't render as javascript:. Same
  stale-response guards on success + catch paths as the other detail drawers.
- Consumes deriveShortageEvidenceQuality for client-side trust indicator
  (three-dot ●●● / ●●○ / ●○○), NOT for severity — severity is classifier
  output.
- Registered in ENERGY_PANELS + panel-layout.ts + components barrel.

Shared store:
- src/shared/fuel-shortage-registry-store.ts — same drain-once memoize
  pattern as pipeline- and storage-facility-registry-store. Both the
  panel and the DeckGL shortage-pins layer read through it.

DeckGL layer:
- DeckGLMap.createEnergyShortagePinsLayer: ScatterplotLayer placing one
  pin per active shortage at the country centroid (via getCountryCentroid
  from services/country-geometry). Stacking offset (~0.8° lon) when
  multiple shortages share a country so Nigeria's petrol + diesel don't
  render as a single dot. Confirmed pins 55km radius; watch 38km. Click
  dispatches 'energy:open-fuel-shortage-detail' — panel listens.
- Tooltip branch on fuel-shortages-layer: country · product · short
  description · severity.
- Layer registered in LAYER_REGISTRY, VARIANT_LAYER_ORDER.energy,
  ENERGY_MAP_LAYERS, ENERGY_MOBILE_MAP_LAYERS. MapLayers.fuelShortages
  is optional on the type so other variants' literals remain valid.

Tests:
- tests/fuel-shortage-registry-store.test.mts (4 tests): drain-once,
  no-data, RPC back-prop, pure-RPC-first path.
- tests/bootstrap.test.mjs — fuelShortages removed from PENDING_CONSUMERS.

Typecheck + typecheck:api clean. 39 tests pass (plus full suite in pre-push).
…rawer

Ships the energy:disruptions:v1 registry that threads together pipelines
and storage facilities: state transitions (sabotage, sanction, maintenance,
mechanical, weather, commercial, war) keyed by assetId so any asset's
drawer can render its history without a second registry lookup.

Data + seeder:
- scripts/data/energy-disruptions.json — 12 curated events spanning
  Nord Stream 1/2 sabotage, Druzhba sanctions, CPC force majeure,
  TurkStream maintenance, Yamal halt, Rehden trusteeship, Arctic LNG 2
  sanction, ESPO drone strikes, BTC fire (historical), Sabine Pass
  Hurricane Beryl, Power of Siberia ramp. Each event links back to a
  seeded asset.
- scripts/_energy-disruption-registry.mjs — validator enforces valid
  assetType/eventType/cause enums, http(s) sources, startAt ≤ endAt,
  MIN_EVENTS=8.
- scripts/seed-energy-disruptions.mjs — runSeed entry (weekly cron).
- Bundle entry at 7×DAY cadence.

Bootstrap 4-file registration (cache-keys.ts + bootstrap.js + health.js +
seed-health.js) — energyDisruptions in PENDING_CONSUMERS because panel
drawers fetch lazily via RPC on drawer-open rather than hydrating from
bootstrap directly.

Proto + handler:
- list_energy_disruptions.proto: ListEnergyDisruptions with
  assetId / assetType / ongoingOnly query facets. Returns events sorted
  newest-first.
- list-energy-disruptions.ts projects raw → wire format, supports all
  three query facets.
- Registered in handler.ts. gateway.ts: 'medium' cache tier.

Shared timeline helper:
- src/shared/disruption-timeline.ts — pure formatters (formatEventWindow,
  formatCapacityOffline, statusForEvent). No generated-type deps so
  PipelineStatusPanel + StorageFacilityMapPanel import the same helpers
  and render the timeline identically.

Panel integration:
- PipelineStatusPanel.loadDetail now fetches getPipelineDetail +
  listEnergyDisruptions({assetId, assetType:'pipeline'}) in parallel.
  Drawer gains "Disruption timeline (N)" section with event type, date
  window, capacity offline, cause chain, and short description per entry.
- StorageFacilityMapPanel gets identical treatment with assetType='storage'.
- Both reset detailEvents on closeDetail and on fresh click (stale-response
  safety).

Tests:
- tests/energy-disruptions-registry.test.mts (17 tests): schema, identity,
  enum coverage, evidence, negative inputs.
- tests/bootstrap.test.mjs — energyDisruptions added to PENDING_CONSUMERS.

Typecheck + typecheck:api clean. 51 tests pass locally (plus full suite
in pre-push).
Extends CountryDeepDivePanel's existing "Energy Profile" card with a
mini Atlas-exposure section that surfaces per-country exposure to the
new registries we shipped in Days 7-13.

For each country:
- Pipelines touching this country (from, to, or transit) — clickable
  rows that dispatch 'energy:open-pipeline-detail' so the PipelineStatusPanel
  drawer opens on the energy variant; no-op on other variants.
- Storage facilities in this country — same loose-coupling pattern
  with 'energy:open-storage-facility-detail'.
- Active fuel shortages in this country — severity breakdown line
  (N confirmed · M watch) plus clickable rows emitting
  'energy:open-fuel-shortage-detail'.

Silent absence: sections render only when the country has matching
assets/events, so countries with no pipeline, storage, or shortage
touchpoints see the existing energy-profile card unchanged.

Lazy stores: reads go through the same shared drain-once stores
(getCachedPipelineRegistries, getCachedStorageFacilityRegistry,
getCachedFuelShortageRegistry) so CountryDeepDivePanel does NOT race
with Atlas panels over the single-drain bootstrap cache. Dynamic
import() keeps the three stores out of the panel's static import graph
so non-energy variants can tree-shake them.

Typecheck clean. No schema changes; purely additive UI read from
already-shipped registries.
Fills the /docs/methodology/disruptions URL referenced by
list_energy_disruptions.proto, scripts/_energy-disruption-registry.mjs,
and the panel attribution footers. Explains scope (state transitions
not daily noise), data shape, what counts as a disruption, classifier
evolution path, RPC contract, and ties into the sibling pipeline +
storage + shortage methodology pages.

No code change; pure docs completion for Week 4 launch polish.
Two handlers (list-storage-facilities + list-pipelines) conflated "empty
filter result on a healthy registry" with "upstream unavailable". A
caller who queried one facilityType/commodityType and legitimately got
zero matches was told the upstream was down — which may push clients to
error-state rendering or suppress caching instead of showing a valid
empty list.

list-storage-facilities.ts — upstreamUnavailable now only fires when
`raw` is null (Redis miss). Zero filtered rows on a healthy registry
returns upstreamUnavailable: false + empty array. Matches the sibling
list-fuel-shortages handler and the wire contract in
list_storage_facilities.proto.

list-pipelines.ts — same bug, subtler shape. Now checks "requested at
least one side AND received nothing" rather than "zero rows after
collection". A filter that legitimately matches no gas/oil pipelines on
a healthy registry now returns upstreamUnavailable: false.

list-energy-disruptions.ts and list-fuel-shortages.ts already had the
correct shape (only flag unavailable when raw is missing) — left as-is.

Typecheck + typecheck:api clean. No tests added: the existing registry
schema tests cover the projection/filter helpers, and the handler-level
gating change is documented in code comments for future audits.
@koala73 koala73 changed the title feat(energy): pipeline registries (gas + oil) — Week 2 Day 6 feat(energy): Energy Atlas end-to-end — pipelines + storage + shortages + disruptions + country drill-down Apr 23, 2026
@koala73
koala73 marked this pull request as ready for review April 23, 2026 03:15
@greptile-apps

greptile-apps Bot commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR delivers the full Energy Atlas feature end-to-end: five Redis registries (gas/oil pipelines, storage facilities, fuel shortages, disruptions), nine RPCs, three DeckGL map layers, three UI panels, and a country drill-down exposure section, all gated behind the energy variant. Architecture is well-considered — the drain-once registry stores solve the single-use bootstrap race, the shared deriver pattern guarantees badge parity between first-paint and post-RPC renders, and the upstreamUnavailable contract is now correctly tied to Redis returning null rather than to filtered-row count.

  • Resolved shortages shown as active pinscreateEnergyShortagePinsLayer and CountryDeepDivePanel.renderAtlasExposure both read all shortage entries without filtering on resolvedAt. This causes resolved shortages to appear as coloured map pins and inflates confirmed/watch counts in the country drill-down, showing stale crisis data as ongoing.
  • upstreamUnavailable inconsistency in disruptions handlerlist-energy-disruptions.ts gates on !raw?.events (fires on partial write) while the rest of the sibling handlers and the PR's stated invariant only fire on !raw (Redis null).

Confidence Score: 4/5

Safe to merge after fixing the resolved-shortage display bug; the upstreamUnavailable inconsistency is low-risk but worth correcting for contract clarity.

Two P1 findings (resolved shortages shown as active on map and in country drill-down) mean users could see misleading crisis data. Both are one-liner filter additions. The upstreamUnavailable inconsistency is P2. All other 66 files are clean.

src/components/DeckGLMap.ts and src/components/CountryDeepDivePanel.ts (resolved shortage filter), server/worldmonitor/supply-chain/v1/list-energy-disruptions.ts (upstreamUnavailable gate)

Important Files Changed

Filename Overview
server/worldmonitor/supply-chain/v1/list-pipelines.ts ListPipelines RPC — dual gas/oil registry merge, correct upstreamUnavailable semantics, clean projection.
server/worldmonitor/supply-chain/v1/list-energy-disruptions.ts ListEnergyDisruptions RPC — upstreamUnavailable check is !raw?.events rather than !raw, inconsistent with sibling handlers and PR's stated contract.
src/shared/pipeline-evidence.ts Badge deriver — clear rule ordering with freshness guard, versioned, duck-typed for server/client reuse.
src/shared/pipeline-registry-store.ts Drain-once memo store — cleanly solves single-use bootstrap race between DeckGLMap and PipelineStatusPanel.
src/components/DeckGLMap.ts Three new energy DeckGL layers; shortage pins layer renders resolved shortages as active — needs active-only filter.
src/components/CountryDeepDivePanel.ts Atlas exposure section added via async imports; shortage count includes resolved entries, inflating active counts.
src/components/FuelShortagePanel.ts New panel — severity-sorted table, evidence drawer, XSS-safe via escapeHtml/sanitizeUrl throughout.
scripts/_pipeline-registry.mjs Registry validator — thorough per-entry checks including evidence-gate for non-flowing badges.
api/bootstrap.js Five new keys added to BOOTSTRAP_CACHE_KEYS and SLOW_KEYS — matches cache-keys.ts additions.

Sequence Diagram

sequenceDiagram
    participant Seed as Railway Cron Seeder
    participant Redis as Upstash Redis
    participant BS as api/bootstrap.js
    participant Store as Registry Store (memo)
    participant Panel as UI Panel
    participant Map as DeckGLMap Layer
    participant RPC as RPC Handler

    Seed->>Redis: SET energy:pipelines:gas:v1 + seed-meta
    Seed->>Redis: SET energy:fuel-shortages:v1 + seed-meta

    BS->>Redis: GET all SLOW_KEYS (incl. new energy keys)
    BS-->>Panel: bootstrap payload (pipelinesGas, fuelShortages ...)

    Panel->>Store: getCachedPipelineRegistries() [drain-once]
    Map->>Store: getCachedPipelineRegistries() [returns memo - no race]
    Store-->>Panel: {gas, oil} registries
    Store-->>Map: same {gas, oil} registries

    Panel->>Panel: derivePipelinePublicBadge(evidence) -> badge
    Map->>Map: derivePipelinePublicBadge(evidence) -> identical badge

    Panel->>RPC: listPipelines() [background refresh]
    RPC->>Redis: getCachedJson(PIPELINES_GAS_KEY) + getCachedJson(PIPELINES_OIL_KEY)
    Redis-->>RPC: raw registries
    RPC->>RPC: projectPipeline + derivePublicBadge
    RPC-->>Panel: ListPipelinesResponse
    Panel->>Store: setCachedPipelineRegistries(fresh data)
    Map->>Map: re-render with updated store on next cycle
Loading

Reviews (1): Last reviewed commit: "fix(energy): upstreamUnavailable only fi..." | Re-trigger Greptile

window.dispatchEvent(new CustomEvent('energy:open-pipeline-detail', {
detail: { pipelineId: obj.id },
}));
} catch {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Resolved shortages rendered as active pins

createEnergyShortagePinsLayer builds pins from every entry in registry?.shortages, with no check for resolvedAt. A shortage marked resolved (non-empty resolvedAt string) will still appear as a coloured pin on the map, which can mislead users into thinking an already-resolved fuel crisis is ongoing. The same issue exists in CountryDeepDivePanel.renderAtlasExposure where the confirmed/watch severity count also includes resolved shortages.

Add a filter before the .map() step:

Suggested change
} catch {
const rawEntries: RawEntry[] = Object.values(registry?.shortages ?? {})
.filter(raw => !raw.resolvedAt);

const confirmedCount = shortages.filter(s => s.severity === 'confirmed').length;
const severityLine = confirmedCount > 0
? `${confirmedCount} confirmed · ${shortages.length - confirmedCount} watch`
: `${shortages.length} watch`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Resolved shortages inflate country drill-down count

shortages here includes entries with a non-empty resolvedAt, so a country might show "2 confirmed · 0 watch" even when all shortages are resolved. The confirmation count and the rows dispatched on click all mix active and historical records with no visual distinction.

Filter to active-only shortages:

Suggested change
: `${shortages.length} watch`;
const shortages = Object.values(registry?.shortages ?? {})
.filter(s => s.country === iso2 && !s.resolvedAt);

Comment on lines +78 to +84
return {
events: [],
fetchedAt: new Date().toISOString(),
classifierVersion: '',
upstreamUnavailable: true,
};
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 upstreamUnavailable fires on partial-write, not only on null Redis return

The PR description and the sibling handlers (list-storage-facilities.ts checks if (!raw)) establish the contract: upstreamUnavailable = Redis returned nothing. Here, if (!raw?.events) will also fire when Redis returns a valid registry object that simply has no events key — e.g. a race during the seed write, or a future schema migration that renames the field. That's a partial write, not a Redis outage, and callers would be sent into error-state rendering unnecessarily.

For consistency with the stated contract, consider splitting the check:

if (!raw) {
  return { events: [], fetchedAt: new Date().toISOString(), classifierVersion: '', upstreamUnavailable: true };
}
// raw.events absent means unexpected seed format — return empty but healthy
const events = Object.values(raw.events ?? {});

Two P1 filter bugs (resolved shortages rendered as active) and one P2
contract inconsistency on the disruptions handler.

P1: DeckGLMap createEnergyShortagePinsLayer rendered every shortage in
the registry as an active crisis pin — including entries where the
classifier has written resolvedAt to mark the crisis over. Added a
filter so only entries with a null/empty resolvedAt become map pins.
Curated v1 data has resolvedAt=null everywhere so no visible change
today, but the moment the classifier starts writing resolutions
post-launch, resolved shortages would have appeared as ongoing.

P1: CountryDeepDivePanel renderAtlasExposure had the same bug in the
country drill-down — "N confirmed · M watch" counts included resolved
entries, inflating the active-crisis line per country. Same one-line
filter fix.

P2: list-energy-disruptions.ts gated upstreamUnavailable on
`!raw?.events` — a partial write (top-level object present but `events`
property missing) fired the "upstream down" flag, inconsistent with
the sibling handlers (list-pipelines, list-storage-facilities,
list-fuel-shortages) that only fire on `!raw`. Rewrote to match:
`!raw` → upstreamUnavailable, empty events → normal empty list. This
also aligns with the contract documented on the upstream-unavailable-
vs-empty-filter skill extracted from the earlier P2 review.

Typecheck + typecheck:api clean. All three fixes are one-liner filter
or gate changes; no test additions needed (registry tests still pass
with v1 data since resolvedAt is null throughout).
@koala73
koala73 merged commit 84ee2be into main Apr 23, 2026
11 checks passed
@koala73
koala73 deleted the feat/energy-atlas-pipelines branch April 23, 2026 03:34
koala73 added a commit that referenced this pull request Apr 23, 2026
…t stdout (#3320)

* chore(bundle-runner): emit reliable per-section summary line on parent stdout

Fixes observability asymmetry in Railway bundle service logs where some
seeders appeared to skip lines like \`Run ID\`, \`Mode\`, or the
structured \`seed_complete\` JSON event. Root cause is Railway's log
ingestion dropping child-stdout lines when multiple seeders emit at
similar timestamps — observed in the PR #3294 launch run where
Pipelines-Gas was missing its \`=== Seed ===\` banner, Pipelines-Oil had
\`Key:\` emitted BEFORE the banner, Storage-Facilities and Energy-
Disruptions were missing Run ID + Mode + seed_complete entirely,
despite identical code paths.

All child processes emit the same lines; Railway just loses some. Fix
is to piggy-back on the observation that bundle-level lines (\`[Bundle:X]
Starting\`, \`[Bundle:X] Finished\`) ARE reliably captured — they come
from the parent process's single stdout stream.

Changes in scripts/_bundle-runner.mjs:
- spawnSeed now captures the child's \`{"event":"seed_complete",...}\` JSON
  line while streaming stdout, parses it, and attaches to the settle
  result.
- Main loop emits one bundle-level summary line per section after child
  exit:
    [Bundle:X] section=NAME status=OK durationMs=1237 records=15 state=OK
  (or \`status=FAILED elapsed=...s reason=...\` for failures).
- Summary line survives Railway's log ingestion even when per-section
  child lines drop, giving monitors a reliable event to key off.

Observability consumers (log-based alerts, seed telemetry scrapers)
should now key off the bundle-level summary rather than per-section
child lines which remain best-effort. The per-section child lines stay
as-is for interactive debugging.

Verification: parse logic sanity-checked against the exact seed_complete
line format. Node syntax check clean. No schema changes.

* fix(bundle-runner): emit FAILED summary line to stderr, not stdout

The prior commit introduced a bundle-level structured summary line per
section. On success that correctly goes to stdout; on FAILED it was
also going to stdout — but that broke tests/bundle-runner.test.mjs test
140 ("timeout emits terminal reason BEFORE SIGTERM/SIGKILL grace").

The test concatenates stdout+stderr and asserts that `SIGKILL` appears
AFTER `Failed after` in the combined string (verifying the kill-decision
log line is emitted BEFORE the 10s SIGTERM→SIGKILL grace window, so
it survives container termination). My new FAILED summary line — which
includes the reason string `timeout after 1s (signal SIGKILL)` —
landed on stdout, which comes first in the concatenation, and its
`SIGKILL` substring matched before the stderr-side `Did not exit on
SIGTERM...SIGKILL` line. Ordering assertion failed.

Fix: route the FAILED summary line through console.error (same stream
as the pre-kill `Failed after ... sending SIGTERM` and the grace-window
`Did not exit...SIGKILL` lines). Chronological ordering in combined
output is preserved; test passes.

OK summary lines stay on stdout — they're observability data, not
error diagnostics, and belong on the normal stream alongside the
bundle Starting/Finished lines.

Local: `node --test tests/bundle-runner.test.mjs` — 4/4 pass including
the previously-failing ordering test.
koala73 added a commit that referenced this pull request Apr 23, 2026
All five methodology pages reference /corrections (the auto-revision-log
URL promised in the Global Energy Flow parity plan §20) but the page
didn't exist — clicks 404'd. This lands the page.

Content:
- Explains the revision-log shape: `{date, assetOrEventId, fieldChanged,
  previousValue, newValue, trigger, sourcesUsed, classifierVersion}`.
- Defines the trigger vocabulary (classifier / source / decay / override)
  so readers know what kind of change they're seeing.
- States the v1-launch truth honestly: the log is empty at launch and
  fills as the post-launch classifier pass (in proactive-intelligence.mjs)
  runs on its normal schedule. No fake entries, no placeholder rows.
- Documents the correction-submission path (operators / regulators /
  researchers with public source URLs) and the contract that
  corrections write `override`-trigger entries citing the submitted
  source — not anonymous overrides.
- Cross-links all five methodology pages.
- Explains WHY we publish this: evidence-first classification only
  works if the audit trail is public; otherwise "the classifier said
  so" has no more authority than any other opaque pipeline.

Also fixes a navigation gap: docs/docs.json was missing both
methodology/disruptions (landed in PR #3294 but never registered in
nav) and the new corrections page. Both now appear in the "Intelligence
& Analysis" group alongside the other methodology pages.

No code changes. MDX lint + docs.json JSON validation pass.
koala73 added a commit that referenced this pull request Apr 23, 2026
…nt) (#3323)

* docs(energy): /corrections revision-log page (Week 4 launch requirement)

All five methodology pages reference /corrections (the auto-revision-log
URL promised in the Global Energy Flow parity plan §20) but the page
didn't exist — clicks 404'd. This lands the page.

Content:
- Explains the revision-log shape: `{date, assetOrEventId, fieldChanged,
  previousValue, newValue, trigger, sourcesUsed, classifierVersion}`.
- Defines the trigger vocabulary (classifier / source / decay / override)
  so readers know what kind of change they're seeing.
- States the v1-launch truth honestly: the log is empty at launch and
  fills as the post-launch classifier pass (in proactive-intelligence.mjs)
  runs on its normal schedule. No fake entries, no placeholder rows.
- Documents the correction-submission path (operators / regulators /
  researchers with public source URLs) and the contract that
  corrections write `override`-trigger entries citing the submitted
  source — not anonymous overrides.
- Cross-links all five methodology pages.
- Explains WHY we publish this: evidence-first classification only
  works if the audit trail is public; otherwise "the classifier said
  so" has no more authority than any other opaque pipeline.

Also fixes a navigation gap: docs/docs.json was missing both
methodology/disruptions (landed in PR #3294 but never registered in
nav) and the new corrections page. Both now appear in the "Intelligence
& Analysis" group alongside the other methodology pages.

No code changes. MDX lint + docs.json JSON validation pass.

* docs(energy): reframe /corrections as planned-surface spec (P1 review fix)

Greptile P1: the prior /corrections page made live-product claims
("writes an append-only entry here", "expect the first entries within
days", "email [email protected]") that the code doesn't
back. The revision-log writer ships with the post-launch classifier;
the correction-intake pipeline does not yet exist; and the related
detail handlers still return empty `revisions` arrays with code
comments explicitly marking the surface as future work.

Fix: rewrite the page as a planned-surface specification with a
prominent Status callout. Changes:

- Page title: "Revision Log" → "Revision Log (Planned)"
- Prominent <Note> callout at the top states v1 launch truth: log is
  not yet live, RPC `revisions` arrays are empty by design,
  corrections are handled manually today.
- "Current state (v1 launch)" section removed; replaced with two
  explicit sections: "What IS live today" (evidence bundles,
  methodology, versioned classifier output) and "What is NOT live
  today" (log entries, automated correction intake, override-writer).
- "Within days" timeline language removed — no false operational SLA.
- Email submission path removed (no automated intake exists). Points
  readers to GitHub issues for manual review today.
- Preserves the planned data shape, trigger vocabulary, policy
  commitment, and "why we publish this" framing — those are spec, not
  claims.

Also softens /corrections references in the four methodology pages
(pipelines, storage, shortages, disruptions) so none of them claim
the revision log is live. Each now says "planned revision-log shape
and submission policy" and points manual corrections at GitHub issues.

MDX lint 122/122 clean. docs.json JSON validation clean. No code
changes; pure reframing to match reality.

* docs(shortages): fix P1 overclaim + wrong RPC name (round-2 review)

Two findings on the same file:

P1 — `energy_asset_overrides` table documented as existing. It doesn't.
The PR's corrections.mdx explicitly lists the override-writer as NOT
live in v1; this section contradicted that. Rewrote as "Break-glass
overrides (planned)" with a clear Status callout matching the pattern
established in docs/corrections.mdx and the other methodology pages.
Points readers at GitHub issues for manual corrections today.

P2 — Wrong RPC name: `listActiveFuelShortages` doesn't exist. The
shipped RPC (in proto/worldmonitor/supply_chain/v1/
list_fuel_shortages.proto + server/worldmonitor/supply-chain/v1/
list-fuel-shortages.ts) is `ListFuelShortages`. Replaced the name +
reframed the sentence to describe what the actual RPC already exposes
(every FuelShortageEntry includes evidence.evidenceSources[]) rather
than projecting a future surface.

Also swept the other methodology pages for the same class of bug:
- grep for _overrides: only the one line above
- grep for listActive/ getActive RPC names: none found
- verified all RPC mentions in docs/methodology + docs/corrections.mdx
  match names actually in proto (ListPipelines, ListStorageFacilities,
  ListFuelShortages, ListEnergyDisruptions, GetPipelineDetail,
  GetStorageFacilityDetail, GetFuelShortageDetail)

MDX lint clean. No code changes.

* docs(methodology): round-3 sibling sweep for revision-log overclaims

Reviewer (Greptile) caught a third round of the same overclaim pattern
I've been trying to stamp out: docs/methodology/shortages.mdx line 46
said "Stale shortages never persist silently. Every demotion writes to
the public revision log." — contradicting the same PR's /corrections
page which explicitly frames the revision log as not-yet-live. Fixed
that one AND did the mechanical sibling sweep the review pattern
clearly called for.

Changes:

- `docs/methodology/shortages.mdx:46` — rewrote the auto-decay footer
  to future tense: "When the post-launch classifier ships, stale
  shortages will never persist silently — every demotion will write
  an entry to the planned public revision log." Points readers at
  /corrections for the designed shape. Notes that today the demotion
  thresholds ARE the contract; the structured audit trail is what
  lands with the classifier.

- `docs/methodology/chokepoints.mdx:64` — sibling sweep caught the
  same bug class ("Every badge transition writes to the public
  revision log"). Reworded to future tense and pointed manual
  corrections at GitHub issues, matching the pattern already applied
  to pipelines / storage / shortages in prior commits on this PR.

Final audit of remaining revision-log mentions across all 5
methodology pages + corrections.mdx — every one uses hedged tense now
("planned", "will", "when live", "designed", "not yet live", "once
the classifier ships"). The one remaining present-tense "emit" in
shortages.mdx:77 is inside the "(planned)" break-glass section with
its own Status callout, so it's correctly scoped.

Following the plan-doc-as-docs-source-overclaim skill's step-4
(sibling sweep) explicitly this time — which also retroactively
validates the skill extraction: three review rounds was the cost of
not running the sweep on round 1.

MDX lint clean. No code changes.

* docs(corrections): drop hardcoded launch date (Greptile P2)

Greptile inline P2 at docs/corrections.mdx:60: the phrase
"v1 launch (2026-04-23)" pins a specific calendar date that will read
inaccurately to visitors months later once entries start appearing.

Dropped the parenthetical date. "Status — v1 launch:" keeps the
scoping clear without tying it to a specific day. When live entries
start appearing on this page (or when the page is rewritten to show
real rows), a "last updated" marker will replace the status callout
entirely — no migration churn needed.

MDX lint 122/122 clean.
koala73 added a commit that referenced this pull request Apr 24, 2026
* feat(energy-atlas): expose Atlas panels on FULL variant + CMD+K

Three Atlas panels (PipelineStatusPanel, StorageFacilityMapPanel,
FuelShortagePanel) shipped in PR #3294 but were registered only in
ENERGY_PANELS — invisible on worldmonitor.app because the energy
variant subdomain is not yet wired. Additionally, no CMD+K entries
existed for them, so command-palette search for "pipeline" or
"storage" returned nothing.

Changes:
- src/config/commands.ts: add panel:pipeline-status,
  panel:storage-facility-map, panel:fuel-shortages with relevant
  keywords (oil/gas/nord stream/druzhba/spr/lng/ugs/rationing/…).
- src/config/panels.ts: add the 3 panel keys to FULL_PANELS with
  priority 2 so they appear in the main worldmonitor.app drawer
  under the existing energy-crisis block. ENERGY_PANELS keeps its
  own priority-1 copies so the future energy.worldmonitor.app
  subdomain still surfaces them top of list.

Unblocks the plan's announcement gate item "UI: at least one Atlas
panel renders registry data on worldmonitor.app in a browser."

Part of docs/internal/energy-atlas-registry-expansion.md follow-up.

* fix(cmd-k): resolve spr/lng keyword collisions so Atlas panel wins

Reviewer found that the original PR wiring let CMD+K "spr" and "lng"
route to the wrong panel because matchCommands() (SearchModal.ts:273)
ranks by exact/prefix/substring then keeps array-insertion order on
ties. Storage-atlas was declared AFTER the colliding entries.

Collisions:
- "spr": panel:oil-inventories (line 105) had exact 'spr' → tied
  with the new storage-facility-map (line 108) → insertion order
  kept oil-inventories winning.
- "lng": panel:hormuz-tracker (line 135) has exact 'lng' →
  storage-facility-map only had substring 'lng terminals' (score 1)
  → hormuz won outright.

Fix:
- Remove 'spr' from oil-inventories keywords. The SPR as a *site
  list* semantically belongs to Strategic Storage Atlas. Stock-level
  queries still route to oil-inventories via 'strategic petroleum'
  (the word 'spr' is not a substring of 'strategic petroleum', so
  no fallback score leaks).
- Add exact 'lng' to storage-facility-map. Both it and hormuz-tracker
  now score 3 on 'lng'; stable sort preserves declaration order,
  so storage (line 108) outranks hormuz (line 135). Hormuz still
  matches via 'hormuz', 'strait of hormuz', 'tanker', 'shipping'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant