Add pipeline flow disruption signals#5
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
koala73
added a commit
that referenced
this pull request
Jan 18, 2026
… freshness Assessment & Documentation: - Add docs/GEOPOLITICAL_ASSESSMENT.md with full platform analysis - Strategic improvement roadmap with prioritized recommendations Quick Win #1 - Data Freshness (intelligence gaps): - Add getIntelligenceGaps() and getIntelligenceGapSummary() to data-freshness.ts - Add human-readable messages explaining what analysts CAN'T see - Add hasCriticalGaps() for alert integration Quick Win #2 - Escalation Scores: - Add escalationScore (1-5), escalationTrend, escalationIndicators to Hotspot type - Update 11 major hotspots with scores (Sahel, Haiti, Horn of Africa, Moscow, Beijing, Kyiv, Taipei, Tehran, Tel Aviv, Pyongyang, Sana'a) Quick Win #3 - Signal Context ("Why It Matters"): - Add SIGNAL_CONTEXT with whyItMatters, actionableInsight, confidenceNote - Add getSignalContext() helper for all 10 signal types - Explains analytical significance of each signal type Quick Win #4 - Historical Context: - Add HistoricalContext interface with lastMajorEvent, precedentCount, cyclicalRisk fields - Add whyItMatters field to Hotspot type - Update major hotspots with historical precedents and geopolitical significance Quick Win #5 - Propaganda Risk Flags: - Add PropagandaRisk type and SourceRiskProfile interface - Add SOURCE_PROPAGANDA_RISK mapping for state media (Xinhua, TASS, RT, CGTN) - Add getSourcePropagandaRisk() and isStateAffiliatedSource() helpers - Flag medium-risk state-affiliated sources (Al Jazeera, France 24, DW, etc.)
facusturla
pushed a commit
to facusturla/worldmonitor
that referenced
this pull request
Feb 27, 2026
… freshness Assessment & Documentation: - Add docs/GEOPOLITICAL_ASSESSMENT.md with full platform analysis - Strategic improvement roadmap with prioritized recommendations Quick Win koala73#1 - Data Freshness (intelligence gaps): - Add getIntelligenceGaps() and getIntelligenceGapSummary() to data-freshness.ts - Add human-readable messages explaining what analysts CAN'T see - Add hasCriticalGaps() for alert integration Quick Win koala73#2 - Escalation Scores: - Add escalationScore (1-5), escalationTrend, escalationIndicators to Hotspot type - Update 11 major hotspots with scores (Sahel, Haiti, Horn of Africa, Moscow, Beijing, Kyiv, Taipei, Tehran, Tel Aviv, Pyongyang, Sana'a) Quick Win koala73#3 - Signal Context ("Why It Matters"): - Add SIGNAL_CONTEXT with whyItMatters, actionableInsight, confidenceNote - Add getSignalContext() helper for all 10 signal types - Explains analytical significance of each signal type Quick Win koala73#4 - Historical Context: - Add HistoricalContext interface with lastMajorEvent, precedentCount, cyclicalRisk fields - Add whyItMatters field to Hotspot type - Update major hotspots with historical precedents and geopolitical significance Quick Win koala73#5 - Propaganda Risk Flags: - Add PropagandaRisk type and SourceRiskProfile interface - Add SOURCE_PROPAGANDA_RISK mapping for state media (Xinhua, TASS, RT, CGTN) - Add getSourcePropagandaRisk() and isStateAffiliatedSource() helpers - Flag medium-risk state-affiliated sources (Al Jazeera, France 24, DW, etc.)
EleCor79
added a commit
to EleCor79/BehavioralHealthPulse
that referenced
this pull request
Mar 2, 2026
Distribute all 32 feeds from feeds-health-italy-eu.csv into the FEEDS config consumed by loadNews() / fetchCategoryFeeds(): - ministero-salute: +MinSalute News Specifiche (CSV koala73#18) - iss-epicentro: +ISS Notizie (CSV koala73#3), +Epicentro Coronavirus (CSV koala73#17) - aifa-tracker: +AIFA Feed (CSV koala73#5), +EMA News (CSV koala73#8), +FDA (CSV koala73#12) - agenas-ospedali: +AGENAS RSS (CSV koala73#4), +PNRR (CSV koala73#6/koala73#19), +Lombardia (CSV koala73#20), +Lazio (CSV koala73#21) - ema-europa: +EMA Clinical Trials (CSV koala73#22) - ecdc-sorveglianza:+ECDC Weekly Threats (CSV koala73#23), +WHO DON (CSV koala73#9), +ProMED (CSV koala73#10) - live-news: +Sanitainformazione (CSV koala73#24), +Humanitas (CSV koala73#26) - europe: +EU Core Health Indicators (CSV koala73#31), +GLOBSEC HRI (CSV koala73#32), +ISTAT (CSV koala73#13), +EIN Health Europe (CSV koala73#29) - rare-diseases: NEW category — EURORDIS (CSV koala73#14), Orphanet IT (CSV koala73#15), Telethon (CSV koala73#16), CDC FluView (CSV koala73#11) Panel disabled by default, available in settings Co-Authored-By: Claude Opus 4.6 <[email protected]>
bradleybond512
referenced
this pull request
in bradleybond512/worldmonitor-macos
Mar 2, 2026
…#5) - Sidecar: add /api/ollama-stream SSE endpoint that calls Ollama with stream:true using Node http.request (bypasses arrayBuffer buffering); returns {skipped:true} JSON when OLLAMA_API_URL not configured - Panel: try streaming first; on SSE response render tokens progressively with blinking cursor and Stop button; fall back to full provider chain if Ollama not configured - CSS: add panel-ai-stop button styles + panel-ai-text--streaming cursor animation (blinking block cursor via ::after pseudo-element) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
gefa-sc
pushed a commit
to gefa-sc/worldmonitor
that referenced
this pull request
Mar 2, 2026
Add mobile responsive design with hamburger menu
20 tasks
This was referenced Mar 9, 2026
3 tasks
SuleymanZeynal
pushed a commit
to SuleymanZeynal/worldmonitor
that referenced
this pull request
Apr 20, 2026
Minerals tab: - Add Boron (Turkey ~70% global share — dominant position) - Add Chromium (Turkey top-5 global producer) - Add Gold (Turkey koala73#5, Azerbaijan growing via Chovdar/Gedabek) - Add Crude Oil/Caspian (Azerbaijan, Kazakhstan, Turkmenistan) Chokepoints: - Bosphorus: add BTC-Ceyhan, Black Sea grain, Caspian oil route IDs Scenarios: - Turkish Straits Closure (Bosphorus blockade — BTC crude + Black Sea grain) - BTC Pipeline Disruption (Baku–Tbilisi–Ceyhan sabotage scenario) https://claude.ai/code/session_01X564hh6CvFxiUhg9DSuJAP
15 tasks
SebastienMelki
added a commit
that referenced
this pull request
Apr 21, 2026
Closes out the remaining @koala73 review findings from #3242 that didn't already land in the HIGH-fix commits, plus the requested CI check that would have caught HIGH #1 (dead-code policy key) at review time. ### MEDIUM #5 — Turnstile missing-secret policy default Flip `verifyTurnstile`'s default `missingSecretPolicy` from `'allow'` to `'allow-in-development'`. Dev with no secret = pass (expected local); prod with no secret = reject + log. submit-contact was already explicitly overriding to `'allow-in-development'`; register-interest was silently getting `'allow'`. Safe default now means a future missing-secret misconfiguration in prod gets caught instead of silently letting bots through. Removed the now-redundant override in submit-contact. ### MEDIUM #6 — Silent enum fallbacks in maritime client `toDisruptionEvent` mapped `AIS_DISRUPTION_TYPE_UNSPECIFIED` / unknown enum values → `gap_spike` / `low` silently. Refactored to return null when either enum is unknown; caller filters nulls out of the array. Handler doesn't produce UNSPECIFIED today, but the `gap_spike` default would have mislabeled the first new enum value the proto ever adds — dropping unknowns is safer than shipping wrong labels. ### LOW — Copy drift in register-interest email Email template hardcoded `435+ Sources`; PR #3241 bumped marketing to `500+`. Bumped in the rewritten file to stay consistent. The `as any` on Convex mutation names carried over from legacy and filed as follow-up #3253. ### Rate-limit-policy coverage lint `scripts/enforce-rate-limit-policies.mjs` validates every key in `ENDPOINT_RATE_POLICIES` resolves to a proto-generated gateway route by cross-referencing `docs/api/*.openapi.yaml`. Fails with the sanctions-entity-search incident referenced in the error message so future drift has a paper trail. Wired into package.json (`lint:rate-limit-policies`) and the pre-push hook alongside `lint:boundaries`. Smoke-tested both directions — clean repo passes (5 policies / 175 routes), seeded drift (the exact HIGH #1 typo) fails with the advertised remedy text. ### Verified - `lint:rate-limit-policies` ✓ - `typecheck` + `typecheck:api` ✓ - `lint:api-contract` ✓ (56 entries) - `lint:boundaries` ✓ - edge-functions + contact-handler tests (147 pass) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
SebastienMelki
added a commit
that referenced
this pull request
Apr 22, 2026
… (#3242) * chore(api): enforce sebuf contract via exceptions manifest (#3207) Adds api/api-route-exceptions.json as the single source of truth for non-proto /api/ endpoints, with scripts/enforce-sebuf-api-contract.mjs gating every PR via npm run lint:api-contract. Fixes the root-only blind spot in the prior allowlist (tests/edge-functions.test.mjs), which only scanned top-level *.js files and missed nested paths and .ts endpoints — the gap that let api/supply-chain/v1/country-products.ts and friends drift under proto domain URL prefixes unchallenged. Checks both directions: every api/<domain>/v<N>/[rpc].ts must pair with a generated service_server.ts (so a deleted proto fails CI), and every generated service must have an HTTP gateway (no orphaned generated code). Manifest entries require category + reason + owner, with removal_issue mandatory for temporary categories (deferred, migration-pending) and forbidden for permanent ones. .github/CODEOWNERS pins the manifest to @SebastienMelki so new exceptions don't slip through review. The manifest only shrinks: migration-pending entries (19 today) will be removed as subsequent commits in this PR land each migration. * refactor(maritime): migrate /api/ais-snapshot → maritime/v1.GetVesselSnapshot (#3207) The proto VesselSnapshot was carrying density + disruptions but the frontend also needed sequence, relay status, and candidate_reports to drive the position-callback system. Those only lived on the raw relay passthrough, so the client had to keep hitting /api/ais-snapshot whenever callbacks were registered and fall back to the proto RPC only when the relay URL was gone. This commit pushes all three missing fields through the proto contract and collapses the dual-fetch-path into one proto client call. Proto changes (proto/worldmonitor/maritime/v1/): - VesselSnapshot gains sequence, status, candidate_reports. - GetVesselSnapshotRequest gains include_candidates (query: include_candidates). Handler (server/worldmonitor/maritime/v1/get-vessel-snapshot.ts): - Forwards include_candidates to ?candidates=... on the relay. - Separate 5-min in-memory caches for the candidates=on and candidates=off variants; they have very different payload sizes and should not share a slot. - Per-request in-flight dedup preserved per-variant. Frontend (src/services/maritime/index.ts): - fetchSnapshotPayload now calls MaritimeServiceClient.getVesselSnapshot directly with includeCandidates threaded through. The raw-relay path, SNAPSHOT_PROXY_URL, DIRECT_RAILWAY_SNAPSHOT_URL and LOCAL_SNAPSHOT_FALLBACK are gone — production already routed via Vercel, the "direct" branch only ever fired on localhost, and the proto gateway covers both. - New toLegacyCandidateReport helper mirrors toDensityZone/toDisruptionEvent. api/ais-snapshot.js deleted; manifest entry removed. Only reduced the codegen scope to worldmonitor.maritime.v1 (buf generate --path) — regenerating the full tree drops // @ts-nocheck from every client/server file and surfaces pre-existing type errors across 30+ unrelated services, which is not in scope for this PR. Shape-diff vs legacy payload: - disruptions / density: proto carries the same fields, just with the GeoCoordinates wrapper and enum strings (remapped client-side via existing toDisruptionEvent / toDensityZone helpers). - sequence, status.{connected,vessels,messages}: now populated from the proto response — was hardcoded to 0/false in the prior proto fallback. - candidateReports: same shape; optional numeric fields come through as 0 instead of undefined, which the legacy consumer already handled. * refactor(sanctions): migrate /api/sanctions-entity-search → LookupSanctionEntity (#3207) The proto docstring already claimed "OFAC + OpenSanctions" coverage but the handler only fuzzy-matched a local OFAC Redis index — narrower than the legacy /api/sanctions-entity-search, which proxied OpenSanctions live (the source advertised in docs/api-proxies.mdx). Deleting the legacy without expanding the handler would have been a silent coverage regression for external consumers. Handler changes (server/worldmonitor/sanctions/v1/lookup-entity.ts): - Primary path: live search against api.opensanctions.org/search/default with an 8s timeout and the same User-Agent the legacy edge fn used. - Fallback path: the existing OFAC local fuzzy match, kept intact for when OpenSanctions is unreachable / rate-limiting. - Response source field flips between 'opensanctions' (happy path) and 'ofac' (fallback) so clients can tell which index answered. - Query validation tightened: rejects q > 200 chars (matches legacy cap). Rate limiting: - Added /api/sanctions/v1/lookup-entity to ENDPOINT_RATE_POLICIES at 30/min per IP — matches the legacy createIpRateLimiter budget. The gateway already enforces per-endpoint policies via checkEndpointRateLimit. Docs: - docs/api-proxies.mdx — dropped the /api/sanctions-entity-search row (plus the orphaned /api/ais-snapshot row left over from the previous commit in this PR). - docs/panels/sanctions-pressure.mdx — points at the new RPC URL and describes the OpenSanctions-primary / OFAC-fallback semantics. api/sanctions-entity-search.js deleted; manifest entry removed. * refactor(military): migrate /api/military-flights → ListMilitaryFlights (#3207) Legacy /api/military-flights read a pre-baked Redis blob written by the seed-military-flights cron and returned flights in a flat app-friendly shape (lat/lon, lowercase enums, lastSeenMs). The proto RPC takes a bbox, fetches OpenSky live, classifies server-side, and returns nested GeoCoordinates + MILITARY_*_TYPE_* enum strings + lastSeenAt — same data, different contract. fetchFromRedis in src/services/military-flights.ts was doing nothing sebuf-aware. Renamed it to fetchViaProto and rewrote to: - Instantiate MilitaryServiceClient against getRpcBaseUrl(). - Iterate MILITARY_QUERY_REGIONS (PACIFIC + WESTERN) in parallel — same regions the desktop OpenSky path and the seed cron already use, so dashboard coverage tracks the analytic pipeline. - Dedup by hexCode across regions. - Map proto → app shape via new mapProtoFlight helper plus three reverse enum maps (AIRCRAFT_TYPE_REVERSE, OPERATOR_REVERSE, CONFIDENCE_REVERSE). The seed cron (scripts/seed-military-flights.mjs) stays put: it feeds regional-snapshot mobility, cross-source signals, correlation, and the health freshness check (api/health.js: 'military:flights:v1'). None of those read the legacy HTTP endpoint; they read the Redis key directly. The proto handler uses its own per-bbox cache keys under the same prefix, so dashboard traffic no longer races the seed cron's blob — the two paths diverge by a small refresh lag, which is acceptable. Docs: dropped the /api/military-flights row from docs/api-proxies.mdx. api/military-flights.js deleted; manifest entry removed. Shape-diff vs legacy: - f.location.{latitude,longitude} → f.lat, f.lon - f.aircraftType: MILITARY_AIRCRAFT_TYPE_TANKER → 'tanker' via reverse map - f.operator: MILITARY_OPERATOR_USAF → 'usaf' via reverse map - f.confidence: MILITARY_CONFIDENCE_LOW → 'low' via reverse map - f.lastSeenAt (number) → f.lastSeen (Date) - f.enrichment → f.enriched (with field renames) - Extra fields registration / aircraftModel / origin / destination / firstSeenAt now flow through where proto populates them. * fix(supply-chain): thread includeCandidates through chokepoint status (#3207) Caught by tsconfig.api.json typecheck in the pre-push hook (not covered by the plain tsc --noEmit run that ran before I pushed the ais-snapshot commit). The chokepoint status handler calls getVesselSnapshot internally with a static no-auth request — now required to include the new includeCandidates bool from the proto extension. Passing false: server-internal callers don't need per-vessel reports. * test(maritime): update getVesselSnapshot cache assertions (#3207) The ais-snapshot migration replaced the single cachedSnapshot/cacheTimestamp pair with a per-variant cache so candidates-on and candidates-off payloads don't evict each other. Pre-push hook surfaced that tests/server-handlers still asserted the old variable names. Rewriting the assertions to match the new shape while preserving the invariants they actually guard: - Freshness check against slot TTL. - Cache read before relay call. - Per-slot in-flight dedup. - Stale-serve on relay failure (result ?? slot.snapshot). * chore(proto): restore // @ts-nocheck on regenerated maritime files (#3207) I ran 'buf generate --path worldmonitor/maritime/v1' to scope the proto regen to the one service I was changing (to avoid the toolchain drift that drops @ts-nocheck from 60+ unrelated files — separate issue). But the repo convention is the 'make generate' target, which runs buf and then sed-prepends '// @ts-nocheck' to every generated .ts file. My scoped command skipped the sed step. The proto-check CI enforces the sed output, so the two maritime files need the directive restored. * refactor(enrichment): decomm /api/enrichment/{company,signals} legacy edge fns (#3207) Both endpoints were already ported to IntelligenceService: - getCompanyEnrichment (/api/intelligence/v1/get-company-enrichment) - listCompanySignals (/api/intelligence/v1/list-company-signals) No frontend callers of the legacy /api/enrichment/* paths exist. Removes: - api/enrichment/company.js, signals.js, _domain.js - api-route-exceptions.json migration-pending entries (58 remain) - docs/api-proxies.mdx rows for /api/enrichment/{company,signals} - docs/architecture.mdx reference updated to the IntelligenceService RPCs Verified: typecheck, typecheck:api, lint:api-contract (89 files / 58 entries), lint:boundaries, tests/edge-functions.test.mjs (136 pass), tests/enrichment-caching.test.mjs (14 pass — still guards the intelligence/v1 handlers), make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(leads): migrate /api/{contact,register-interest} → LeadsService (#3207) New leads/v1 sebuf service with two POST RPCs: - SubmitContact → /api/leads/v1/submit-contact - RegisterInterest → /api/leads/v1/register-interest Handler logic ported 1:1 from api/contact.js + api/register-interest.js: - Turnstile verification (desktop sources bypass, preserved) - Honeypot (website field) silently accepts without upstream calls - Free-email-domain gate on SubmitContact (422 ApiError) - validateEmail (disposable/offensive/typo-TLD/MX) on RegisterInterest - Convex writes via ConvexHttpClient (contactMessages:submit, registerInterest:register) - Resend notification + confirmation emails (HTML templates unchanged) Shared helpers moved to server/_shared/: - turnstile.ts (getClientIp + verifyTurnstile) - email-validation.ts (disposable/offensive/MX checks) Rate limits preserved via ENDPOINT_RATE_POLICIES: - submit-contact: 3/hour per IP (was in-memory 3/hr) - register-interest: 5/hour per IP (was in-memory 5/hr; desktop sources previously capped at 2/hr via shared in-memory map — now 5/hr like everyone else, accepting the small regression in exchange for Upstash-backed global limiting) Callers updated: - pro-test/src/App.tsx contact form → new submit-contact path - src-tauri/sidecar/local-api-server.mjs cloud-fallback rewrites /api/register-interest → /api/leads/v1/register-interest when proxying; keeps local path for older desktop builds - src/services/runtime.ts isKeyFreeApiTarget allows both old and new paths through the WORLDMONITOR_API_KEY-optional gate Tests: - tests/contact-handler.test.mjs rewritten to call submitContact handler directly; asserts on ValidationError / ApiError - tests/email-validation.test.mjs + tests/turnstile.test.mjs point at the new server/_shared/ modules Deleted: api/contact.js, api/register-interest.js, api/_ip-rate-limit.js, api/_turnstile.js, api/_email-validation.js, api/_turnstile.test.mjs. Manifest entries removed (58 → 56). Docs updated (api-platform, api-commerce, usage-rate-limits). Verified: npm run typecheck + typecheck:api + lint:api-contract (88 files / 56 entries) + lint:boundaries pass; full test:data (5852 tests) passes; make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * chore(pro-test): rebuild bundle for leads/v1 contact form (#3207) Updates the enterprise contact form to POST to /api/leads/v1/submit-contact (old path /api/contact removed in the previous commit). Bundle is rebuilt from pro-test/src/App.tsx source change in 9ccd309. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review): address HIGH review findings 1-3 (#3207) Three review findings from @koala73 on the sebuf-migration PR, all silent bugs that would have shipped to prod: ### 1. Sanctions rate-limit policy was dead code ENDPOINT_RATE_POLICIES keyed the 30/min budget under /api/sanctions/v1/lookup-entity, but the generated route (from the proto RPC LookupSanctionEntity) is /api/sanctions/v1/lookup-sanction-entity. hasEndpointRatePolicy / getEndpointRatelimit are exact-string pathname lookups, so the mismatch meant the endpoint fell through to the generic 600/min global limiter instead of the advertised 30/min. Net effect: the live OpenSanctions proxy endpoint (unauthenticated, external upstream) had 20x the intended rate budget. Fixed by renaming the policy key to match the generated route. ### 2. Lost stale-seed fallback on military-flights Legacy api/military-flights.js cascaded military:flights:v1 → military:flights:stale:v1 before returning empty. The new proto handler went straight to live OpenSky/relay and returned null on miss. Relay or OpenSky hiccup used to serve stale seeded data (24h TTL); under the new handler it showed an empty map. Both keys are still written by scripts/seed-military-flights.mjs on every run — fix just reads the stale key when the live fetch returns null, converts the seed's app-shape flights (flat lat/lon, lowercase enums, lastSeenMs) to the proto shape (nested GeoCoordinates, enum strings, lastSeenAt), and filters to the request bbox. Read via getRawJson (unprefixed) to match the seed cron's writes, which bypass the env-prefix system. ### 3. Hex-code casing mismatch broke getFlightByHex The seed cron writes hexCode: icao24.toUpperCase() (uppercase); src/services/military-flights.ts:getFlightByHex uppercases the lookup input: f.hexCode === hexCode.toUpperCase(). The new proto handler preserved OpenSky's lowercase icao24, and mapProtoFlight is a pass-through. getFlightByHex was silently returning undefined for every call after the migration. Fix: uppercase in the proto handler (live + stale paths), and document the invariant in a comment on MilitaryFlight.hex_code in military_flight.proto so future handlers don't re-break it. ### Verified - typecheck + typecheck:api clean - lint:api-contract (56 entries) / lint:boundaries clean - tests/edge-functions.test.mjs 130 pass - make generate zero-diff (openapi spec regenerated for proto comment) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review): restore desktop 2/hr rate cap on register-interest (#3207) Addresses HIGH review finding #4 from @koala73. The legacy api/register-interest.js applied a nested 2/hr per-IP cap when `source === 'desktop-settings'`, on top of the generic 5/hr endpoint budget. The sebuf migration lost this — desktop-source requests now enjoy the full 5/hr cap. Since `source` is an unsigned client-supplied field, anyone sending `source: 'desktop-settings'` skips Turnstile AND gets 5/hr. Without the tighter cap the Turnstile bypass is cheaper to abuse. Added `checkScopedRateLimit` to `server/_shared/rate-limit.ts` — a reusable second-stage Upstash limiter keyed on an opaque scope string + caller identifier. Fail-open on Redis errors to match existing checkRateLimit / checkEndpointRateLimit semantics. Handlers that need per-subscope caps on top of the gateway-level endpoint budget use this helper. In register-interest: when `isDesktopSource`, call checkScopedRateLimit with scope `/api/leads/v1/register-interest#desktop`, limit=2, window=1h, IP as identifier. On exceeded → throw ApiError(429). ### What this does not fix This caps the blast radius of the Turnstile bypass but does not close it — an attacker sending `source: 'desktop-settings'` still skips Turnstile (just at 2/hr instead of 5/hr). The proper fix is a signed desktop-secret header that authenticates the bypass; filed as follow-up #3252. That requires coordinated Tauri build + Vercel env changes out of scope for #3207. ### Verified - typecheck + typecheck:api clean - lint:api-contract (56 entries) - tests/edge-functions.test.mjs + contact-handler.test.mjs (147 pass) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review): MEDIUM + LOW + rate-limit-policy CI check (#3207) Closes out the remaining @koala73 review findings from #3242 that didn't already land in the HIGH-fix commits, plus the requested CI check that would have caught HIGH #1 (dead-code policy key) at review time. ### MEDIUM #5 — Turnstile missing-secret policy default Flip `verifyTurnstile`'s default `missingSecretPolicy` from `'allow'` to `'allow-in-development'`. Dev with no secret = pass (expected local); prod with no secret = reject + log. submit-contact was already explicitly overriding to `'allow-in-development'`; register-interest was silently getting `'allow'`. Safe default now means a future missing-secret misconfiguration in prod gets caught instead of silently letting bots through. Removed the now-redundant override in submit-contact. ### MEDIUM #6 — Silent enum fallbacks in maritime client `toDisruptionEvent` mapped `AIS_DISRUPTION_TYPE_UNSPECIFIED` / unknown enum values → `gap_spike` / `low` silently. Refactored to return null when either enum is unknown; caller filters nulls out of the array. Handler doesn't produce UNSPECIFIED today, but the `gap_spike` default would have mislabeled the first new enum value the proto ever adds — dropping unknowns is safer than shipping wrong labels. ### LOW — Copy drift in register-interest email Email template hardcoded `435+ Sources`; PR #3241 bumped marketing to `500+`. Bumped in the rewritten file to stay consistent. The `as any` on Convex mutation names carried over from legacy and filed as follow-up #3253. ### Rate-limit-policy coverage lint `scripts/enforce-rate-limit-policies.mjs` validates every key in `ENDPOINT_RATE_POLICIES` resolves to a proto-generated gateway route by cross-referencing `docs/api/*.openapi.yaml`. Fails with the sanctions-entity-search incident referenced in the error message so future drift has a paper trail. Wired into package.json (`lint:rate-limit-policies`) and the pre-push hook alongside `lint:boundaries`. Smoke-tested both directions — clean repo passes (5 policies / 175 routes), seeded drift (the exact HIGH #1 typo) fails with the advertised remedy text. ### Verified - `lint:rate-limit-policies` ✓ - `typecheck` + `typecheck:api` ✓ - `lint:api-contract` ✓ (56 entries) - `lint:boundaries` ✓ - edge-functions + contact-handler tests (147 pass) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(commit 5): decomm /api/eia/* + migrate /api/satellites → IntelligenceService (#3207) Both targets turned out to be decomm-not-migration cases. The original plan called for two new services (economic/v1.GetEiaSeries + natural/v1.ListSatellitePositions) but research found neither was needed: ### /api/eia/[[...path]].js — pure decomm, zero consumers The "catch-all" is a misnomer — only two paths actually worked, /api/eia/health and /api/eia/petroleum, both Redis-only readers. Zero frontend callers in src/. Zero server-side readers. Nothing consumes the `energy:eia-petroleum:v1` key that seed-eia-petroleum.mjs writes daily. The EIA data the frontend actually uses goes through existing typed RPCs in economic/v1: GetEnergyPrices, GetCrudeInventories, GetNatGasStorage, GetEnergyCapacity. None of those touch /api/eia/*. Building GetEiaSeries would have been dead code. Deleted the legacy file + its test (tests/api-eia-petroleum.test.mjs — it only covered the legacy endpoint, no behavior to preserve). Empty api/eia/ dir removed. **Note for review:** the Redis seed cron keeps running daily and nothing consumes it. If that stays unused, seed-eia-petroleum.mjs should be retired too (separate PR). Out of scope for sebuf-migration. ### /api/satellites.js — Learning #2 strikes again IntelligenceService.ListSatellites already exists at /api/intelligence/v1/list-satellites, reads the same Redis key (intelligence:satellites:tle:v1), and supports an optional country filter the legacy didn't have. One frontend caller in src/services/satellites.ts needed to switch from `fetch(toApiUrl('/api/satellites'))` to the typed IntelligenceServiceClient.listSatellites. Shape diff was tiny — legacy `noradId` became proto `id` (handler line 36 already picks either), everything else identical. alt/velocity/inclination in the proto are ignored by the caller since it propagates positions client-side via satellite.js. Kept the client-side cache + failure cooldown + 20s timeout (still valid concerns at the caller level). ### Manifest + docs - api-route-exceptions.json: 56 → 54 entries (both removed) - docs/api-proxies.mdx: dropped the two rows from the Raw-data passthroughs table ### Verified - typecheck + typecheck:api ✓ - lint:api-contract (54 entries) / lint:boundaries / lint:rate-limit-policies ✓ - tests/edge-functions.test.mjs 127 pass (down from 130 — 3 tests were for the deleted eia endpoint) - make generate zero-diff (no proto changes) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(commit 6): migrate /api/supply-chain/v1/{country-products,multi-sector-cost-shock} → SupplyChainService (#3207) Both endpoints were hand-rolled TS handlers sitting under a proto URL prefix — the exact drift the manifest guardrail flagged. Promoted both to typed RPCs: - GetCountryProducts → /api/supply-chain/v1/get-country-products - GetMultiSectorCostShock → /api/supply-chain/v1/get-multi-sector-cost-shock Handlers preserve the existing semantics: PRO-gate via isCallerPremium(ctx.request), iso2 / chokepointId validation, raw bilateral-hs4 Redis read (skip env-prefix to match seeder writes), CHOKEPOINT_STATUS_KEY for war-risk tier, and the math from _multi-sector-shock.ts unchanged. Empty-data and non-PRO paths return the typed empty payload (no 403 — the sebuf gateway pattern is empty-payload-on-deny). Client wrapper switches from premiumFetch to client.getCountryProducts/ client.getMultiSectorCostShock. Legacy MultiSectorShock / MultiSectorShockResponse / CountryProductsResponse names remain as type aliases of the generated proto types so CountryBriefPanel + CountryDeepDivePanel callsites compile with zero churn. Manifest 54 → 52. Rate-limit gateway routes 175 → 177. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(gateway): add cache-tier entries for new supply-chain RPCs (#3207) Pre-push tests/route-cache-tier.test.mjs caught the missing entries. Both PRO-gated, request-varying — match the existing supply-chain PRO cohort (get-country-cost-shock, get-bypass-options, etc.) at slow-browser tier. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(commit 7): migrate /api/scenario/v1/{run,status,templates} → ScenarioService (#3207) Promote the three literal-filename scenario endpoints to a typed sebuf service with three RPCs: POST /api/scenario/v1/run-scenario (RunScenario) GET /api/scenario/v1/get-scenario-status (GetScenarioStatus) GET /api/scenario/v1/list-scenario-templates (ListScenarioTemplates) Preserves all security invariants from the legacy handlers: - 405 for wrong method (sebuf service-config method gate) - scenarioId validation against SCENARIO_TEMPLATES registry - iso2 regex ^[A-Z]{2}$ - JOB_ID_RE path-traversal guard on status - Per-IP 10/min rate limit (moved to gateway ENDPOINT_RATE_POLICIES) - Queue-depth backpressure (>100 → 429) - PRO gating via isCallerPremium - AbortSignal.timeout on every Redis pipeline (runRedisPipeline helper) Wire-level diffs vs legacy: - Per-user RL now enforced at the gateway (same 10/min/IP budget). - Rate-limit response omits Retry-After header; retryAfter is in the body per error-mapper.ts convention. - ListScenarioTemplates emits affectedHs2: [] when the registry entry is null (all-sectors sentinel); proto repeated cannot carry null. - RunScenario returns { jobId, status } (no statusUrl field — unused by SupplyChainPanel, drop from wire). Gateway wiring: - server/gateway.ts RPC_CACHE_TIER: list-scenario-templates → 'daily' (matches legacy max-age=3600); get-scenario-status → 'slow-browser' (premium short-circuit target, explicit entry required by tests/route-cache-tier.test.mjs). - src/shared/premium-paths.ts: swap old run/status for the new run-scenario/get-scenario-status paths. - api/scenario/v1/{run,status,templates}.ts deleted; 3 manifest exceptions removed (63 → 52 → 49 migration-pending). Client: - src/services/scenario/index.ts — typed client wrapper using premiumFetch (injects Clerk bearer / API key). - src/components/SupplyChainPanel.ts — polling loop swapped from premiumFetch strings to runScenario/getScenarioStatus. Hard 20s timeout on run preserved via AbortSignal.any. Tests: - tests/scenario-handler.test.mjs — 18 new handler-level tests covering every security invariant + the worker envelope coercion. - tests/edge-functions.test.mjs — scenario sections removed, replaced with a breadcrumb pointer to the new test file. Docs: api-scenarios.mdx, scenario-engine.mdx, usage-rate-limits.mdx, usage-errors.mdx, supply-chain.mdx refreshed with new paths. Verified: typecheck, typecheck:api, lint:api-contract (49 entries), lint:rate-limit-policies (6/180), lint:boundaries, route-cache-tier (parity), full edge-functions (117) + scenario-handler (18). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(commit 8): migrate /api/v2/shipping/{route-intelligence,webhooks} → ShippingV2Service (#3207) Partner-facing endpoints promoted to a typed sebuf service. Wire shape preserved byte-for-byte (camelCase field names, ISO-8601 fetchedAt, the same subscriberId/secret formats, the same SET + SADD + EXPIRE 30-day Redis pipeline). Partner URLs /api/v2/shipping/* are unchanged. RPCs landed: - GET /route-intelligence → RouteIntelligence (PRO, slow-browser) - POST /webhooks → RegisterWebhook (PRO) - GET /webhooks → ListWebhooks (PRO, slow-browser) The existing path-parameter URLs remain on the legacy edge-function layout because sebuf's HTTP annotations don't currently model path params (grep proto/**/*.proto for `path: "{…}"` returns zero). Those endpoints are split into two Vercel dynamic-route files under api/v2/shipping/webhooks/, behaviorally identical to the previous hybrid file but cleanly separated: - GET /webhooks/{subscriberId} → [subscriberId].ts - POST /webhooks/{subscriberId}/rotate-secret → [subscriberId]/[action].ts - POST /webhooks/{subscriberId}/reactivate → [subscriberId]/[action].ts Both get manifest entries under `migration-pending` pointing at #3207. Other changes - scripts/enforce-sebuf-api-contract.mjs: extended GATEWAY_RE to accept api/v{N}/{domain}/[rpc].ts (version-first) alongside the canonical api/{domain}/v{N}/[rpc].ts; first-use of the reversed ordering is shipping/v2 because that's the partner contract. - vite.config.ts: dev-server sebuf interceptor regex extended to match both layouts; shipping/v2 import + allRoutes entry added. - server/gateway.ts: RPC_CACHE_TIER entries for /api/v2/shipping/ route-intelligence + /webhooks (slow-browser; premium-gated endpoints short-circuit to slow-browser but the entries are required by tests/route-cache-tier.test.mjs). - src/shared/premium-paths.ts: route-intelligence + webhooks added. - tests/shipping-v2-handler.test.mjs: 18 handler-level tests covering PRO gate, iso2/cargoType/hs2 coercion, SSRF guards (http://, RFC1918, cloud metadata, IMDS), chokepoint whitelist, alertThreshold range, secret/subscriberId format, pipeline shape + 30-day TTL, cross-tenant owner isolation, `secret` omission from list response. Manifest delta - Removed: api/v2/shipping/route-intelligence.ts, api/v2/shipping/webhooks.ts - Added: api/v2/shipping/webhooks/[subscriberId].ts (migration-pending) - Added: api/v2/shipping/webhooks/[subscriberId]/[action].ts (migration-pending) - Added: api/internal/brief-why-matters.ts (internal-helper) — regression surface from the #3248 main merge, which introduced the file without a manifest entry. Filed here to keep the lint green; not strictly in scope for commit 8 but unblocking. Net result: 49 → 47 `migration-pending` entries (one net-removal even though webhook path-params stay pending, because two files collapsed into two dynamic routes). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review HIGH 1): SupplyChainServiceClient must use premiumFetch (#3207) Signed-in browser pro users were silently hitting 401 on 8 supply-chain premium endpoints (country-products, multi-sector-cost-shock, country-chokepoint-index, bypass-options, country-cost-shock, sector-dependency, route-explorer-lane, route-impact). The shared client was constructed with globalThis.fetch, so no Clerk bearer or X-WorldMonitor-Key was injected. The gateway's validateApiKey runs with forceKey=true for PREMIUM_RPC_PATHS and 401s before isCallerPremium is consulted. The generated client's try/catch collapses the 401 into an empty-fallback return, leaving panels blank with no visible error. Fix is one line at the client constructor: swap globalThis.fetch for premiumFetch. The same pattern is already in use for insider-transactions, stock-analysis, stock-backtest, scenario, trade (premiumClient) — this was an omission on this client, not a new pattern. premiumFetch no-ops safely when no credentials are available, so the 5 non-premium methods on this client (shippingRates, chokepointStatus, chokepointHistory, criticalMinerals, shippingStress) continue to work unchanged. This also fixes two panels that were pre-existing latently broken on main (chokepoint-index, bypass-options, etc. — predating #3207, not regressions from it). Commit 6 expanded the surface by routing two more methods through the same buggy client; this commit fixes the class. From koala73 review (#3242 second-pass, HIGH new #1): > Exact class PR #3233 fixed for RegionalIntelligenceBoard / > DeductionPanel / trade / country-intel. Supply-chain was not in > #3233's scope. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review HIGH 2): restore 400 on input-shape errors for 2 supply-chain handlers (#3207) Commit 6 collapsed all non-happy paths into empty-200 on `get-country-products` and `get-multi-sector-cost-shock`, including caller-bug cases that legacy returned 400 for: - get-country-products: malformed iso2 → empty 200 (was 400) - get-multi-sector-cost-shock: malformed iso2 / missing chokepointId / unknown chokepointId → empty 200 (was 400) The commit message for 6 called out the 403-for-non-pro → empty-200 shift ("sebuf gateway pattern is empty-payload-on-deny") but not the 400 shift. They're different classes: - Empty-payload-200 for PRO-deny: intentional contract change, already documented and applied across the service. Generated clients treat "you lack PRO" as "no data" — fine. - Empty-payload-200 for malformed input: caller bug silently masked. External API consumers can't distinguish "bad wiring" from "genuinely no data", test harnesses lose the signal, bad calling code doesn't surface in Sentry. Fix: `throw new ValidationError(violations)` on the 3 input-shape branches. The generated sebuf server maps ValidationError → HTTP 400 (see src/generated/server/.../service_server.ts and leads/v1 which already uses this pattern). PRO-gate deny stays as empty-200 — that contract shift was intentional and is preserved. Regression tests added at tests/supply-chain-validation.test.mjs (8 cases) pinning the three-way contract: - bad input → 400 (ValidationError) - PRO-gate deny on valid input → 200 empty - valid PRO input, no data in Redis → 200 empty (unchanged) From koala73 review (#3242 second-pass, HIGH new #2). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review HIGH 3): restore statusUrl on RunScenarioResponse + document 202→200 wire break (#3207) Commit 7 silently shifted /api/scenario/v1/run-scenario's response contract in two ways that the commit message covered only partially: 1. HTTP 202 Accepted → HTTP 200 OK 2. Dropped `statusUrl` string from the response body The `statusUrl` drop was mentioned as "unused by SupplyChainPanel" but not framed as a contract change. The 202 → 200 shift was not mentioned at all. This is a same-version (v1 → v1) migration, so external callers that key off either signal — `response.status === 202` or `response.body.statusUrl` — silently branch incorrectly. Evaluated options: (a) sebuf per-RPC status-code config — not available. sebuf's HttpConfig only models `path` and `method`; no status annotation. (b) Bump to scenario/v2 — judged heavier than the break itself for a single status-code shift. No in-repo caller uses 202 or statusUrl; the docs-level impact is containable. (c) Accept the break, document explicitly, partially restore. Took option (c): - Restored `statusUrl` in the proto (new field `string status_url = 3` on RunScenarioResponse). Server computes `/api/scenario/v1/get-scenario-status?jobId=<encoded job_id>` and populates it on every successful enqueue. External callers that followed this URL keep working unchanged. - 202 → 200 is not recoverable inside the sebuf generator, so it is called out explicitly in two places: - docs/api-scenarios.mdx now includes a prominent `<Warning>` block documenting the v1→v1 contract shift + the suggested migration (branch on response body shape, not HTTP status). - RunScenarioResponse proto comment explains why 200 is the new success status on enqueue. OpenAPI bundle regenerated to reflect the restored statusUrl field. - Regression test added in tests/scenario-handler.test.mjs pinning `statusUrl` to the exact URL-encoded shape — locks the invariant so a future proto rename or handler refactor can't silently drop it again. From koala73 review (#3242 second-pass, HIGH new #3). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review HIGH 1/2): close webhook tenant-isolation gap on shipping/v2 (#3207) Koala flagged this as a merge blocker in PR #3242 review. server/worldmonitor/shipping/v2/{register-webhook,list-webhooks}.ts migrated without reinstating validateApiKey(req, { forceKey: true }), diverging from both the sibling api/v2/shipping/webhooks/[subscriberId] routes and the documented "X-WorldMonitor-Key required" contract in docs/api-shipping-v2.mdx. Attack surface: the gateway accepts Clerk bearer auth as a pro signal. A Clerk-authenticated pro user with no X-WorldMonitor-Key reaches the handler, callerFingerprint() falls back to 'anon', and every such caller collapses into a shared webhook:owner:anon:v1 bucket. The defense-in-depth ownerTag !== ownerHash check in list-webhooks.ts doesn't catch it because both sides equal 'anon' — every Clerk-session holder could enumerate / overwrite every other Clerk-session pro tenant's registered webhook URLs. Fix: reinstate validateApiKey(ctx.request, { forceKey: true }) at the top of each handler, throwing ApiError(401) when absent. Matches the sibling routes exactly and the published partner contract. Tests: - tests/shipping-v2-handler.test.mjs: two existing "non-PRO → 403" tests for register/list were using makeCtx() with no key, which now fails at the 401 layer first. Renamed to "no API key → 401 (tenant-isolation gate)" with a comment explaining the failure mode being tested. 18/18 pass. Verified: typecheck:api, lint:api-contract (no change), lint:boundaries, lint:rate-limit-policies, test:data (6005/6005). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix(review HIGH 2/2): restore v1 path aliases on scenario + supply-chain (#3207) Koala flagged this as a merge blocker in PR #3242 review. Commits 6 + 7 of #3207 renamed five documented v1 URLs to the sebuf method-derived paths and deleted the legacy edge-function files: POST /api/scenario/v1/run → run-scenario GET /api/scenario/v1/status → get-scenario-status GET /api/scenario/v1/templates → list-scenario-templates GET /api/supply-chain/v1/country-products → get-country-products GET /api/supply-chain/v1/multi-sector-cost-shock → get-multi-sector-cost-shock server/router.ts is an exact static-match table (Map keyed on `METHOD PATH`), so any external caller — docs, partner scripts, grep-the- internet — hitting the old documented URL would 404 on first request after merge. Commit 8 (shipping/v2) preserved partner URLs byte-for- byte; the scenario + supply-chain renames missed that discipline. Fix: add five thin alias edge functions that rewrite the pathname to the canonical sebuf path and delegate to the domain [rpc].ts gateway via a new server/alias-rewrite.ts helper. Premium gating, rate limits, entitlement checks, and cache-tier lookups all fire on the canonical path — aliases are pure URL rewrites, not a duplicate handler pipeline. api/scenario/v1/{run,status,templates}.ts api/supply-chain/v1/{country-products,multi-sector-cost-shock}.ts Vite dev parity: file-based routing at api/ is a Vercel concern, so the dev middleware (vite.config.ts) gets a matching V1_ALIASES rewrite map before the router dispatch. Manifest: 5 new entries under `deferred` with removal_issue=#3282 (tracking their retirement at the next v1→v2 break). lint:api-contract stays green (89 files checked, 55 manifest entries validated). Docs: - docs/api-scenarios.mdx: migration callout at the top with the full old→new URL table and a link to the retirement issue. - CHANGELOG.md + docs/changelog.mdx: Changed entry documenting the rename + alias compat + the 202→200 shift (from commit 23c821a). Verified: typecheck:api, lint:api-contract, lint:rate-limit-policies, lint:boundaries, test:data (6005/6005). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
koala73
added a commit
that referenced
this pull request
Apr 22, 2026
…tirements + §3.6 coverage gate Methodology-doc update capturing the three §3.5 landings and the §3.6 CI gate. Five edits: 1. **Known construct limitations section (#5 and #6):** strikethrough the original "dead signals" and "no coverage-based weight cap" items, annotate them with "Landed in PR 3 §3.5"/"Landed in PR 3 §3.6" + specifics of what shipped. 2. **Currency & External H4 section:** completely rewritten. Old table (fxVolatility / fxDeviation / fxReservesAdequacy on BIS primary) is replaced by the two-indicator post-PR-3 table (inflationStability at 0.60 + fxReservesAdequacy at 0.40). Coverage ladder spelled out (0.85 / 0.55 / 0.40 / 0.30). Legacy BIS indicators named as experimental-tier drill-downs only. 3. **Fuel Stock Days H4 section:** H4 heading text kept verbatim so the methodology-lint H4-to-dimension mapping does not break; body rewritten to explain that the dimension is retired from core but the seeder still runs for IEA-member drill-downs. 4. **External Debt Coverage table row:** goalpost 5-0 → 2-0, description cites Greenspan-Guidotti reserve-adequacy rule. 5. **New v2.2 changelog entry** — PR 3 dead-signal cleanup, covering §3.5 points 1/2/3 + §3.6 + acceptance gates + construct-audit updates. No scoring or code changes in this commit. Methodology-lint test passes (H4 mapping intact). All 6327 data-tier tests pass.
koala73
added a commit
that referenced
this pull request
Apr 22, 2026
* feat(resilience): PR 3 §3.5 — retire fuelStockDays from core score permanently
First commit in PR 3 of the resilience repair plan. Retires
`fuelStockDays` from the core score with no replacement.
Why permanent, not replaced:
IEA emergency-stockholding rules are defined in days of NET IMPORTS
and do not bind net exporters by design. Norway/Canada/US measured
in days-of-imports are incomparable to Germany/Japan measured the
same way — the construct is fundamentally different across the two
country classes. No globally-comparable recovery-fuel signal can
be built from this source; the pre-repair probe showed 100% imputed
at 50 for every country in the April 2026 freeze.
scoreFuelStockDays:
- Rewritten to return coverage=0 + observedWeight=0 +
imputationClass='source-failure' for every country regardless
of seed content.
- Drops the dimension from the `recovery` domain's coverage-
weighted mean automatically; remaining recovery dimensions
pick up the share via re-normalisation in
`_shared.ts#coverageWeightedMean`.
- No explicit weight transfer needed — the coverage-weighted
blend handles redistribution.
Registry:
- recoveryFuelStockDays re-tagged from tier='enrichment' to
tier='experimental' so the Core coverage gate treats it as
out-of-score.
- Description updated to make the retirement explicit; entry
stays in the registry for structural continuity (the
dimension `fuelStockDays` remains in RESILIENCE_DIMENSION_ORDER
for the 19-dimension tests; removing the dimension entirely is
a PR 4 structural-audit concern).
Housekeeping:
- Removed `RESILIENCE_RECOVERY_FUEL_STOCKS_KEY` constant (no
longer read; noUnusedLocals would reject it).
- Removed `RecoveryFuelStocksCountry` interface for the same
reason. Comment at the removed declaration instructs future
maintainers not to re-add the type as a reservation; when a
new recovery-fuel concept lands, introduce a fresh interface.
Plan reference: §3.5 point 1 of
`docs/plans/2026-04-22-001-fix-resilience-scorer-structural-bias-plan.md`.
51 resilience tests pass, typecheck + biome clean. The
`recovery` domain's published score will shift slightly for every
country because the 0.10 slot that fuelStockDays was imputing to
now redistributes; the compare-harness acceptance-gate rerun at
merge time will quantify the shift per plan §6 gates.
* feat(resilience): PR 3 §3.5 — retire BIS-backed currencyExternal; rebuild on IMF inflation + WB reserves
BIS REER/DSR feeds were load-bearing in currencyExternal (weights 0.35
fxVolatility + 0.35 fxDeviation, ~70% of dimension). They cover ~60
countries max — so every non-BIS country fell through to
curated_list_absent (coverage 0.3) or a thin IMF proxy (coverage 0.45).
Combined with reserveMarginPct already removed in PR 1, currencyExternal
was the clearest "construct absent for most of the world" carrier left
in the scorer.
Changes:
_dimension-scorers.ts
- scoreCurrencyExternal now reads IMF macro (inflationPct) + WB FX
reserves only. Coverage ladder:
inflation + reserves → 0.85 (observed primary + secondary)
inflation only → 0.55
reserves only → 0.40
neither → 0.30 (IMPUTE.bisEer retained for snapshot
continuity; semantics read as
"no IMF + no WB reserves" now)
- Removed dead symbols: RESILIENCE_BIS_EXCHANGE_KEY constant (reserved
via comment only, flagged by noUnusedLocals), stddev() helper,
getCountryBisExchangeRates() loader, BisExchangeRate interface,
dateToSortableNumber() — all were exclusive callers of the retired
BIS path.
_indicator-registry.ts
- New core entry inflationStability (weight 0.60, tier=core,
sourceKey=economic:imf:macro:v2).
- fxReservesAdequacy weight 0.15 → 0.40 (secondary reliability
anchor).
- fxVolatility + fxDeviation demoted tier=enrichment → tier=experimental
(BIS ~60-country coverage; off the core weight sum).
- Non-experimental weights now sum to 1.0 (0.60 + 0.40).
scripts/compare-resilience-current-vs-proposed.mjs
- EXTRACTION_RULES: added inflationStability →
imf-macro-country-field field=inflationPct so the registry-parity
test passes and the correlation harness sees the new construct.
tests/resilience-dimension-scorers.test.mts
- Dropped BIS-era wording ("non-BIS country") and test 266
(BIS-outage coverage 0.35 branch) which collapsed to the inflation-
only path post-retirement.
- Updated coverage assertions: inflation-only 0.45 → 0.55; inflation+
reserves 0.55 → 0.85.
tests/resilience-scorers.test.mts
- domainAverages.economic 68.33 → 66.33 (US currencyExternal score
shifts slightly under IMF+reserves vs old BIS composite).
- stressScore 67.85 → 67.21; stressFactor 0.3215 → 0.3279.
- overallScore 65.82 → 65.52.
- baselineScore unchanged (currencyExternal is stress-only).
All 6324 data-tier tests pass. typecheck:api clean. No change to
seeders or Redis keys; this is a pure scorer + registry rebuild.
* feat(resilience): PR 3 §3.5 point 3 — re-goalpost externalDebtCoverage (0..5 → 0..2)
Plan §2.1 diagnosis table showed externalDebtCoverage saturating at
score=100 across all 9 probe countries — including stressed states.
Signal was collapsed. Root cause: (worst=5, best=0) gave every country
with ratio < 0.5 a score above 90, and mapped Greenspan-Guidotti's
reserve-adequacy threshold (ratio=1.0) to score 80 — well into "no
worry" territory instead of the "mild warning" it should be.
Re-anchored on Greenspan-Guidotti directly: ratio=1.0 now maps to score
50 (mild warning), ratio=2.0 to score 0 (acute rollover-shock exposure).
Ratios above 2.0 clamp to 0, consistent with "beyond this point the
country is already in crisis; exact value stops mattering."
Files changed:
- _indicator-registry.ts: recoveryDebtToReserves goalposts
{worst: 5, best: 0} → {worst: 2, best: 0}. Description updated to
cite Greenspan-Guidotti; inline comment documents anchor + rationale.
- _dimension-scorers.ts: scoreExternalDebtCoverage normalizer bound
changed from (0..5) to (0..2), with inline comment.
- docs/methodology/country-resilience-index.mdx: goalpost table row
5-0 → 2-0, description cites Greenspan-Guidotti.
- docs/methodology/indicator-sources.yaml:
* constructStatus: dead-signal → observed-mechanism (signal is now
discriminating).
* reviewNotes updated to describe the new anchor.
* mechanismTestRationale names the Greenspan-Guidotti rule.
- tests/resilience-dimension-monotonicity.test.mts: updated the
comment + picked values inside the (0..2) discriminating band (0.3
and 1.5). Old values (1 vs 4) had 4 clamping to 0.
- tests/resilience-dimension-scorers.test.mts: NO score threshold
relaxed >90 → >=85 (NO ratio=0.2 now scores 90, was 96).
- tests/resilience-scorers.test.mts: fixture drift:
* domainAverages.recovery 54.83 → 47.33 (US extDebt 70 → 25).
* baselineScore 63.63 → 60.12 (extDebt is baseline type).
* overallScore 65.52 → 63.27.
* stressScore / stressFactor unchanged (extDebt is baseline-only).
All 6324 data-tier tests pass. typecheck:api clean.
* feat(resilience): PR 3 §3.6 — CI gate on indicator coverage and nominal weight
Plan §3.6 adds a new acceptance criterion (also §5 item 5):
> No indicator with observed coverage below 70% may exceed 5% nominal
> weight OR 5% effective influence in the post-change sensitivity run.
This commit enforces the NOMINAL-WEIGHT half as a unit test that runs
on every CI build. The EFFECTIVE-INFLUENCE half is produced by
scripts/validate-resilience-sensitivity.mjs as a committed artifact;
the gate file only asserts that script still exists so a refactor that
removes it breaks the build loudly.
Why the gate exists (plan §3.6):
"A dimension at 30% observed coverage carries the same effective
weight as one at 95%. This contradicts the OECD/JRC handbook on
uncertainty analysis."
Implementation:
tests/resilience-coverage-influence-gate.test.mts — three tests:
1. Nominal-weight gate: for every core indicator with coverage < 137
countries (70% of the ~195-country universe), computes its nominal
overall weight as
indicator.weight × (1/dimensions-in-domain) × domain-weight
and asserts it does not exceed 5%. Equal-share-per-dimension is
the *upper bound* on runtime weight (coverage-weighted mean gives
a lower share when a dimension drops out), so this is a strict
bound: if the nominal number passes, the runtime number also
passes for every country.
2. Effective-influence contract: asserts the sensitivity script
exists at its expected path. Removing it (intentionally or by
refactor) breaks the build.
3. Audit visibility: prints the top 10 core indicators by nominal
overall weight. No assertion beyond "ran" — the list lets
reviewers spot outliers that pass the gate but are near the cap.
Current state (observed from audit output):
recoveryReserveMonths: nominal=4.17% coverage=188
recoveryDebtToReserves: nominal=4.17% coverage=185
recoveryImportHhi: nominal=4.17% coverage=190
inflationStability: nominal=3.40% coverage=185
electricityConsumption: nominal=3.30% coverage=217
ucdpConflict: nominal=3.09% coverage=193
Every core indicator has coverage ≥ 180 (already enforced by the
pre-existing indicator-tiering test), so the nominal-weight gate has
no current violators — its purpose is catching future drift, not
flagging today's state.
All 6327 data-tier tests pass. typecheck:api clean.
* docs(resilience): PR 3 methodology doc — document §3.5 dead-signal retirements + §3.6 coverage gate
Methodology-doc update capturing the three §3.5 landings and the §3.6 CI
gate. Five edits:
1. **Known construct limitations section (#5 and #6):** strikethrough the
original "dead signals" and "no coverage-based weight cap" items,
annotate them with "Landed in PR 3 §3.5"/"Landed in PR 3 §3.6" +
specifics of what shipped.
2. **Currency & External H4 section:** completely rewritten. Old table
(fxVolatility / fxDeviation / fxReservesAdequacy on BIS primary) is
replaced by the two-indicator post-PR-3 table (inflationStability at
0.60 + fxReservesAdequacy at 0.40). Coverage ladder spelled out
(0.85 / 0.55 / 0.40 / 0.30). Legacy BIS indicators named as
experimental-tier drill-downs only.
3. **Fuel Stock Days H4 section:** H4 heading text kept verbatim so the
methodology-lint H4-to-dimension mapping does not break; body
rewritten to explain that the dimension is retired from core but the
seeder still runs for IEA-member drill-downs.
4. **External Debt Coverage table row:** goalpost 5-0 → 2-0, description
cites Greenspan-Guidotti reserve-adequacy rule.
5. **New v2.2 changelog entry** — PR 3 dead-signal cleanup, covering
§3.5 points 1/2/3 + §3.6 + acceptance gates + construct-audit
updates.
No scoring or code changes in this commit. Methodology-lint test passes
(H4 mapping intact). All 6327 data-tier tests pass.
* fix(resilience): PR 3 §3.6 gate — correct share-denominator for coverage-weighted aggregation
Reviewer catch (thanks). The previous gate computed each indicator's
nominal overall weight as
indicator.weight × (1 / N_total_dimensions_in_domain) × domain_weight
and claimed this was an upper bound ("actual runtime weight is ≤ this
when some dimensions drop out on coverage"). That is BACKWARDS for
this scorer.
The domain aggregation is coverage-weighted
(server/worldmonitor/resilience/v1/_shared.ts coverageWeightedMean),
so when a dimension pins at coverage=0 it is EXCLUDED from the
denominator and the surviving dimensions' shares go UP, not down.
PR 3 commit 1 retires fuelStockDays by hard-coding its scorer to
coverage=0 for every country — so in the current live state the
recovery domain has 5 contributing dimensions (not 6), and each core
recovery indicator's nominal share is
1.0 × 1/5 × 0.25 = 5.00% (was mis-reported as 4.17%)
The old gate therefore under-estimated nominal influence and could
silently pass exactly the kind of low-coverage overweight regression
it is meant to block.
Fix:
- Added `coreBearingDimensions(domainId)` helper that counts only
dimensions that have ≥1 core indicator in the registry. A dimension
with only experimental/enrichment entries (post-retirement
fuelStockDays) has no core contribution → does not dilute shares.
- Updated `nominalOverallWeight` to divide by the core-bearing count,
not the raw dimension count.
- Rewrote the helper's doc comment to stop claiming this is a strict
upper bound — explicitly calls out the dynamic case (source failure
raising surviving dim shares further) as the sensitivity script's
responsibility.
- Added a new regression test: asserts (a) at least one recovery
dimension is all-non-core (fuelStockDays post-retirement),
(b) fuelStockDays has zero core indicators, and (c) recoveryDebt
ToReserves nominal = 0.05 exactly (not 0.0417) — any reversion
of the retirement or regression to N_total-denominator will fail
loudly.
Top-10 audit output now correctly shows:
recoveryReserveMonths: nominal=5% coverage=188
recoveryDebtToReserves: nominal=5% coverage=185
recoveryImportHhi: nominal=5% coverage=190
(was 4.17% each under the old math)
All 486 resilience tests pass. typecheck:api clean.
Note: the 5% figure is exactly AT the cap, not over it. "exceed" means
strictly > 5%, so it still passes. But now the reviewer / audit log
reflects reality.
* fix(resilience): PR 3 review — retired-dim confidence drag + false source-failure label
Addresses the Codex review P1 + P2 on PR #3297.
P1 — retired-dim drag on confidence averages
--------------------------------------------
scoreFuelStockDays returns coverage=0 by design (retired construct),
but computeLowConfidence, computeOverallCoverage, and the widget's
formatResilienceConfidence averaged across all 19 dimensions. That
dragged every country's reported averageCoverage down — US went from
0.8556 (active dims only) to 0.8105 (all dims) — enough drift to
misclassify edge countries as lowConfidence and to shift the ranking
widget's overallCoverage pill for every country.
Fix: introduce an authoritative RESILIENCE_RETIRED_DIMENSIONS set in
_dimension-scorers.ts and filter it out of all three averages. The
filter is keyed on the retired-dim REGISTRY, not on coverage === 0,
because a non-retired dim can legitimately emit coverage=0 on a
genuinely sparse-data country via weightedBlend fall-through — those
entries MUST keep dragging confidence down (that is the sparse-data
signal lowConfidence exists to surface). Verified: sparse-country
release-gate test (marks sparse WHO/FAO countries as low confidence)
still passes with the registry-keyed filter; would have failed with
a naive coverage=0 filter.
Server-client parity: widget-utils cannot import server code, so
RESILIENCE_RETIRED_DIMENSION_IDS is a hand-mirrored constant, kept
in lockstep by tests/resilience-retired-dimensions-parity.test.mts
(parses the widget file as text, same pattern as existing widget-util
tests that can't import the widget module directly).
P2 — false "Source down" label on retired dim
---------------------------------------------
scoreFuelStockDays hard-coded imputationClass: 'source-failure',
which the widget maps to "Source down: upstream seeder failed" with
a `!` icon for every country. That is semantically wrong for an
intentional retirement. Flipped to null so the widget's absent-path
renders a neutral cell without a false outage label. null is already
a legal value of ResilienceDimensionScore.imputationClass; no type
change needed.
Tests
-----
- tests/resilience-confidence-averaging.test.mts (new): pins the
registry-keyed filter semantic for computeOverallCoverage +
computeLowConfidence. Includes a negative-control test proving
non-retired coverage=0 dims still flip lowConfidence.
- tests/resilience-retired-dimensions-parity.test.mts (new):
lockstep gate between server and client retired-dim lists.
- Widget test adds a registry-keyed exclusion test with a non-retired
coverage=0 dim in the fixture to lock in the correct semantic.
- Existing tests asserting imputationClass: 'source-failure' for
fuelStockDays flipped to null.
All 494 resilience tests + full 6336/6336 data-tier suite pass.
Typecheck clean for both tsconfig.json and tsconfig.api.json.
* docs(resilience): align methodology + registry metadata with shipped imputationClass=null
Follow-up to the previous PR 3 review commit that flipped
scoreFuelStockDays's imputationClass from 'source-failure' to null to
avoid a false "Source down" widget label on every country. The code
changed; the doc and registry metadata did not, leaving three sites
in the methodology mdx and two comment/description sites in the
registry still claiming imputationClass='source-failure'. Any future
reviewer (or tooling that treats the registry description as
authoritative) would be misled.
This commit rewrites those sites to describe the shipped behavior:
- imputationClass=null (not 'source-failure'), with the rationale
- exclusion from confidence/coverage averages via the
RESILIENCE_RETIRED_DIMENSIONS registry filter
- the distinction between structural retirement (filtered) and
runtime coverage=0 (kept so sparse-data countries still flag
lowConfidence)
Touched:
- docs/methodology/country-resilience-index.mdx (lines ~33, ~268, ~590)
- server/worldmonitor/resilience/v1/_indicator-registry.ts
(recoveryFuelStockDays comment block + description field)
No code-behavior change. Docs-only.
Tests: 157 targeted resilience tests pass (incl. methodology-lint +
widget + release-gate + confidence-averaging). Typecheck clean on
both tsconfig.json and tsconfig.api.json.
4 tasks
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…w (§R #5 = B) Per plan §R/#5 decision B: denormalise countries[] at seed time on each disruption event so CountryDeepDivePanel can filter events per country without an asset-registry round trip. Schema join (pipeline/storage → event.assetId) happens once in the weekly cron, not on every panel render. The alternative (client-side join) was rejected because it couples UI logic to asset-registry internals and duplicates the join for every surface that wants a per-country filter. Changes: - `proto/.../list_energy_disruptions.proto`: add `repeated string countries = 15` to EnergyDisruptionEntry with doc comment tying it to the plan decision and the always-non-empty invariant. - `scripts/_energy-disruption-registry.mjs`: • Load pipeline-gas + pipeline-oil + storage-facilities registries once per seed cycle; index by id. • `deriveCountriesForEvent()` resolves assetId to {fromCountry, toCountry, transitCountries} (pipeline) or {country} (storage), deduped + alpha-sorted so byte-diff stability holds. • `buildPayload()` attaches the computed countries[] to every event before writing. • `validateRegistry()` now requires non-empty countries[] of ISO2 codes. Combined with the seeder's `emptyDataIsFailure: true`, this surfaces orphaned assetIds loudly — the next cron tick fails validation and seed-meta stays stale, tripping health alarms. - `scripts/data/energy-disruptions.json`: fix two orphaned assetIds that the new join caught: • `cpc-force-majeure-2022`: `cpc-pipeline` → `cpc` (matches the entry in pipelines-oil.json). • `pdvsa-designation-2019`: `ve-petrol-2026-q1` (non-existent) → `venezuela-anzoategui-puerto-la-cruz`. - `server/.../list-energy-disruptions.ts`: project countries[] into the RPC response via coerceStringArray. Legacy pre-denorm rows surface as empty array (always present on wire, length 0 => old). - `src/components/CountryDeepDivePanel.ts`: add 4th Atlas row — "Energy disruptions in {iso2}" — filtered by `iso2 ∈ countries[]`. Failure is silent; EnergyDisruptionsPanel (upcoming) is the primary disruption surface. - `tests/energy-disruptions-registry.test.mts`: switch to validating the buildPayload output (post-denorm), add §R #5 B invariant tests, plus a raw-JSON invariant ensuring curators don't hand-edit countries[] (it's derived, not declared). Proto regen note: `make generate` currently fails with a duplicate openapi plugin collision in buf.gen.yaml (unrelated bug — 3 plugin entries emit to the same out dir). Worked around by temporarily trimming buf.gen.yaml to just the TS plugins for this regen. Added only the `countries: string[]` wire field to both service_client and service_server; no other generated-file drift in this PR.
koala73
added a commit
that referenced
this pull request
Apr 24, 2026
…ntryDeepDive row (§R #5 = B) (#3377) * feat(energy-atlas): seed-side countries[] denorm + CountryDeepDive row (§R #5 = B) Per plan §R/#5 decision B: denormalise countries[] at seed time on each disruption event so CountryDeepDivePanel can filter events per country without an asset-registry round trip. Schema join (pipeline/storage → event.assetId) happens once in the weekly cron, not on every panel render. The alternative (client-side join) was rejected because it couples UI logic to asset-registry internals and duplicates the join for every surface that wants a per-country filter. Changes: - `proto/.../list_energy_disruptions.proto`: add `repeated string countries = 15` to EnergyDisruptionEntry with doc comment tying it to the plan decision and the always-non-empty invariant. - `scripts/_energy-disruption-registry.mjs`: • Load pipeline-gas + pipeline-oil + storage-facilities registries once per seed cycle; index by id. • `deriveCountriesForEvent()` resolves assetId to {fromCountry, toCountry, transitCountries} (pipeline) or {country} (storage), deduped + alpha-sorted so byte-diff stability holds. • `buildPayload()` attaches the computed countries[] to every event before writing. • `validateRegistry()` now requires non-empty countries[] of ISO2 codes. Combined with the seeder's `emptyDataIsFailure: true`, this surfaces orphaned assetIds loudly — the next cron tick fails validation and seed-meta stays stale, tripping health alarms. - `scripts/data/energy-disruptions.json`: fix two orphaned assetIds that the new join caught: • `cpc-force-majeure-2022`: `cpc-pipeline` → `cpc` (matches the entry in pipelines-oil.json). • `pdvsa-designation-2019`: `ve-petrol-2026-q1` (non-existent) → `venezuela-anzoategui-puerto-la-cruz`. - `server/.../list-energy-disruptions.ts`: project countries[] into the RPC response via coerceStringArray. Legacy pre-denorm rows surface as empty array (always present on wire, length 0 => old). - `src/components/CountryDeepDivePanel.ts`: add 4th Atlas row — "Energy disruptions in {iso2}" — filtered by `iso2 ∈ countries[]`. Failure is silent; EnergyDisruptionsPanel (upcoming) is the primary disruption surface. - `tests/energy-disruptions-registry.test.mts`: switch to validating the buildPayload output (post-denorm), add §R #5 B invariant tests, plus a raw-JSON invariant ensuring curators don't hand-edit countries[] (it's derived, not declared). Proto regen note: `make generate` currently fails with a duplicate openapi plugin collision in buf.gen.yaml (unrelated bug — 3 plugin entries emit to the same out dir). Worked around by temporarily trimming buf.gen.yaml to just the TS plugins for this regen. Added only the `countries: string[]` wire field to both service_client and service_server; no other generated-file drift in this PR. * chore(proto): regenerate openapi specs for countries[] field Runs `make generate` with the sebuf v0.11.1 plugin now correctly resolved via the PATH fix (cherry-picked from fix/makefile-generate-path-prefix). The new `countries` field on EnergyDisruptionEntry propagates into: - docs/api/SupplyChainService.openapi.yaml (primary per-service spec) - docs/api/SupplyChainService.openapi.json (machine-readable variant) - docs/api/worldmonitor.openapi.yaml (consolidated bundle) No TypeScript drift beyond the already-committed service_client.ts / service_server.ts updates in 80797e7. * fix(energy-atlas): drop highlightEventId emission (review P2) Codex P2: loadDisruptionsForCountry dispatched `highlightEventId` but neither PipelineStatusPanel nor StorageFacilityMapPanel consumes it (the openDetailHandler reads only pipelineId / facilityId). The UI's implicit promise (event-specific highlighting) wasn't delivered — clickthrough was asset-generic, and the extra wire field was a misleading API surface. Fix: emit only {pipelineId, facilityId} in the dispatched detail. Row click opens the asset drawer; user sees the full per-asset disruption timeline and locates the event visually. Symmetric fix for PR #3378's EnergyDisruptionsPanel — both emitters now match the drawer contract exactly. Re-add `highlightEventId` here when the drawer panels ship matching consumer code (openDetailHandler accepts it, loadDetail stores it, renderDisruptionTimeline scrolls + emphasises the matching event). Typecheck clean, test:data 6698/6698 pass. * fix(energy-atlas): collision detection + abort signal + label clamp (review P2) Three Codex P2 findings on PR #3377: 1. `loadAssetRegistries()` spread-merged gas + oil pipelines, silently overwriting entries on id collision. No collision today, but a curator adding a pipeline under the same id to both files would cause `deriveCountriesForEvent` to return wrong-commodity country data with no test flagging it. Fix: explicit merge loop that throws on duplicate id. The next cron tick fails validation, seed-meta stays stale, health alarms fire — same loud-failure pattern the rest of the seeder uses. 2. `loadDisruptionsForCountry` didn't thread `this.signal` through the RPC fetch shim. The stale-closure guard (`currentCode !== iso2`) discarded stale RESULTS, but the in-flight request couldn't be cancelled when the user switched countries or closed the panel. Fix: wrap globalThis.fetch with { signal: this.signal } in the client factory, matching the signal lifecycle the rest of the panel already uses. 3. `shortDescription` values up to 200 chars rendered without ellipsis in the compact Atlas row, overflowing the row layout. Fix: new `truncateDisruptionLabel` helper clamps to 80 chars with ellipsis. Full text still accessible via click-through to the asset drawer. Typecheck clean, test:data 6698/6698 pass.
koala73
added a commit
that referenced
this pull request
Apr 25, 2026
…lead divergence (#3396) * feat(brief-llm): canonical synthesis prompt + v3 cache key Extends generateDigestProse to be the single source of truth for brief executive-summary synthesis (canonicalises what was previously split between brief-llm's generateDigestProse and seed-digest- notifications.mjs's generateAISummary). Ports Brain B's prompt features into buildDigestPrompt: - ctx={profile, greeting, isPublic} parameter (back-compat: 4-arg callers behave like today) - per-story severity uppercased + short-hash prefix [h:XXXX] so the model can emit rankedStoryHashes for stable re-ranking - profile lines + greeting opener appear only when ctx.isPublic !== true validateDigestProseShape gains optional rankedStoryHashes (≥4-char strings, capped to MAX_STORIES_PER_USER × 2). v2-shaped rows still pass — field defaults to []. hashDigestInput v3: - material includes profile-SHA, greeting bucket, isPublic flag, per-story hash - isPublic=true substitutes literal 'public' for userId in the cache key so all share-URL readers of the same (date, sensitivity, pool) hit ONE cache row (no PII in public cache key) Adds generateDigestProsePublic(stories, sensitivity, deps) wrapper — no userId param by design — for the share-URL surface. Cache prefix bumped brief:llm:digest:v2 → v3. v2 rows expire on TTL. Per the v1→v2 precedent (see hashDigestInput comment), one-tick cost on rollout is acceptable for cache-key correctness. Tests: 72/72 passing in tests/brief-llm.test.mjs (8 new for the v3 behaviors), full data suite 6952/6952. Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md Step 1, Codex-approved (5 rounds). * feat(brief): envelope v3 — adds digest.publicLead for share-URL surface Bumps BRIEF_ENVELOPE_VERSION 2 → 3. Adds optional BriefDigest.publicLead — non-personalised executive lead generated by generateDigestProsePublic (already in this branch from the previous commit) for the public share-URL surface. Personalised `lead` is the canonical synthesis for authenticated channels; publicLead is its profile-stripped sibling so api/brief/public/* never serves user-specific content (watched assets/regions). SUPPORTED_ENVELOPE_VERSIONS = [1, 2, 3] keeps v1 + v2 envelopes in the 7-day TTL window readable through the rollout — the composer only ever writes the current version, but readers must tolerate older shapes that haven't expired yet. Same rollout pattern used at the v1 → v2 bump. Renderer changes (server/_shared/brief-render.js): - ALLOWED_DIGEST_KEYS gains 'publicLead' (closed-key-set still enforced; v2 envelopes pass because publicLead === undefined is the v2 shape). - assertBriefEnvelope: new isNonEmptyString check on publicLead when present. Type contract enforced; absence is OK. Tests (tests/brief-magazine-render.test.mjs): - New describe block "v3 publicLead field": v3 envelope renders; malformed publicLead rejected; v2 envelope still passes; ad-hoc digest keys (e.g. synthesisLevel) still rejected — confirming the closed-key-set defense holds for the cron-local-only fields the orchestrator must NOT persist. - BRIEF_ENVELOPE_VERSION pin updated 2 → 3 with rollout-rationale comment. Test results: 182 brief-related tests pass; full data suite 6956/6956. Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md Step 2, Codex Round-3 Medium #2. * feat(brief): synthesis splice + rankedStoryHashes pre-cap re-order Plumbs the canonical synthesis output (lead, threads, signals, publicLead, rankedStoryHashes from generateDigestProse) through the pure composer so the orchestration layer can hand pre-resolved data into envelope.digest. Composer stays sync / no I/O — Codex Round-2 High #2 honored. Changes: scripts/lib/brief-compose.mjs: - digestStoryToUpstreamTopStory now emits `hash` (the digest story's stable identifier, falls back to titleHash when absent). Without this, rankedStoryHashes from the LLM has nothing to match against. - composeBriefFromDigestStories accepts opts.synthesis = {lead, threads, signals, rankedStoryHashes?, publicLead?}. When passed, splices into envelope.digest after the stub is built. Partial synthesis (e.g. only `lead` populated) keeps stub defaults for the other fields — graceful degradation when L2 fallback fires. shared/brief-filter.js: - filterTopStories accepts optional rankedStoryHashes. New helper applyRankedOrder re-orders stories by short-hash prefix match BEFORE the cap is applied, so the model's editorial judgment of importance survives MAX_STORIES_PER_USER. Stable for ties; stories not in the ranking come after in original order. Empty/missing ranking is a no-op (legacy callers unchanged). shared/brief-filter.d.ts: - filterTopStories signature gains rankedStoryHashes?: string[]. - UpstreamTopStory gains hash?: unknown (carried through from digestStoryToUpstreamTopStory). Tests added (tests/brief-from-digest-stories.test.mjs): - synthesis substitutes lead/threads/signals/publicLead. - legacy 4-arg callers (no synthesis) keep stub lead. - partial synthesis (only lead) keeps stub threads/signals. - rankedStoryHashes re-orders pool before cap. - short-hash prefix match (model emits 8 chars; story carries full). - unranked stories go after in original order. Test results: 33/33 in brief-from-digest-stories; 182/182 across all brief tests; full data suite 6956/6956. Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md Step 3, Codex Round-2 Low + Round-2 High #2. * feat(brief): single canonical synthesis per user; rewire all channels Restructures the digest cron's per-user compose + send loops to produce ONE canonical synthesis per user per issueSlot — the lead text every channel (email HTML, plain-text, Telegram, Slack, Discord, webhook) and the magazine show is byte-identical. This eliminates the "two-brain" divergence that was producing different exec summaries on different surfaces (observed 2026-04-25 0802). Architecture: composeBriefsForRun (orchestration): - Pre-annotates every eligible rule with lastSentAt + isDue once, before the per-user pass. Same getLastSentAt helper the send loop uses so compose + send agree on lastSentAt for every rule. composeAndStoreBriefForUser (per-user): - Two-pass winner walk: try DUE rules first (sortedDue), fall back to ALL eligible rules (sortedAll) for compose-only ticks. Preserves today's dashboard refresh contract for weekly / twice_daily users on non-due ticks (Codex Round-4 High #1). - Within each pass, walk by compareRules priority and pick the FIRST candidate with a non-empty pool — mirrors today's behavior at scripts/seed-digest-notifications.mjs:1044 and prevents the "highest-priority but empty pool" edge case (Codex Round-4 Medium #2). - Three-level synthesis fallback chain: L1: generateDigestProse(fullPool, ctx={profile,greeting,!public}) L2: generateDigestProse(envelope-sized slice, ctx={}) L3: stub from assembleStubbedBriefEnvelope Distinct log lines per fallback level so ops can quantify failure-mode distribution. - Generates publicLead in parallel via generateDigestProsePublic (no userId param; cache-shared across all share-URL readers). - Splices synthesis into envelope via composer's optional `synthesis` arg (Step 3); rankedStoryHashes re-orders the pool BEFORE the cap so editorial importance survives MAX_STORIES. - synthesisLevel stored in the cron-local briefByUser entry — NOT persisted in the envelope (renderer's assertNoExtraKeys would reject; Codex Round-2 Medium #5). Send loop: - Reads lastSentAt via shared getLastSentAt helper (single source of truth with compose flow). - briefLead = brief?.envelope?.data?.digest?.lead — the canonical lead. Passed to buildChannelBodies (text/Telegram/Slack/Discord), injectEmailSummary (HTML email), and sendWebhook (webhook payload's `summary` field). All-channel parity (Codex Round-1 Medium #6). - Subject ternary reads cron-local synthesisLevel: 1 or 2 → "Intelligence Brief", 3 → "Digest" (preserves today's UX for fallback paths; Codex Round-1 Missing #5). Removed: - generateAISummary() — the second LLM call that produced the divergent email lead. ~85 lines. - AI_SUMMARY_CACHE_TTL constant — no longer referenced. The digest:ai-summary:v1:* cache rows expire on their existing 1h TTL (no cleanup pass). Helpers added: - getLastSentAt(rule) — extracted Upstash GET for digest:last-sent so compose + send both call one source of truth. - buildSynthesisCtx(rule, nowMs) — formats profile + greeting for the canonical synthesis call. Preserves all today's prefs-fetch failure-mode behavior. Composer: - compareRules now exported from scripts/lib/brief-compose.mjs so the cron can sort each pass identically to groupEligibleRulesByUser. Test results: full data suite 6962/6962 (was 6956 pre-Step 4; +6 new compose-synthesis tests from Step 3). Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md Steps 4 + 4b. Codex-approved (5 rounds). * fix(brief-render): public-share lead fail-safe — never leak personalised lead Public-share render path (api/brief/public/[hash].ts → renderer publicMode=true) MUST NEVER serve the personalised digest.lead because that string can carry profile context — watched assets, saved-region names, etc. — written by generateDigestProse with ctx.profile populated. Previously: redactForPublic redacted user.name and stories.whyMatters but passed digest.lead through unchanged. Codex Round-2 High (security finding). Now (v3 envelope contract): - redactForPublic substitutes digest.lead = digest.publicLead when the v3 envelope carries one (generated by generateDigestProsePublic with profile=null, cache-shared across all public readers). - When publicLead is absent (v2 envelope still in TTL window OR v3 envelope where publicLead generation failed), redactForPublic sets digest.lead to empty string. - renderDigestGreeting: when lead is empty, OMIT the <blockquote> pull-quote entirely. Page still renders complete (greeting + horizontal rule), just without the italic lead block. - NEVER falls back to the original personalised lead. assertBriefEnvelope still validates publicLead's contract (when present, must be a non-empty string) BEFORE redactForPublic runs, so a malformed publicLead throws before any leak risk. Tests added (tests/brief-magazine-render.test.mjs): - v3 envelope renders publicLead in pull-quote, personalised lead text never appears. - v2 envelope (no publicLead) omits pull-quote; rest of page intact. - empty-string publicLead rejected by validator (defensive). - private render still uses personalised lead. Test results: 68 brief-magazine-render tests pass; full data suite remains green from prior commit. Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md Step 5, Codex Round-2 High (security). * feat(digest): brief lead parity log + extra acceptance tests Adds the parity-contract observability line and supplementary acceptance tests for the canonical synthesis path. Parity log (per send, after successful delivery): [digest] brief lead parity user=<id> rule=<v>:<s>:<lang> synthesis_level=<1|2|3> exec_len=<n> brief_lead_len=<n> channels_equal=<bool> public_lead_len=<n> When channels_equal=false an extra WARN line fires — "PARITY REGRESSION user=… — email lead != envelope lead." Sentry's existing console-breadcrumb hook lifts this without an explicit captureMessage call. Plan acceptance criterion A5. Tests added (tests/brief-llm.test.mjs, +9): - generateDigestProsePublic: two distinct callers with identical (sensitivity, story-pool) hit the SAME cache row (per Codex Round-2 Medium #4 — "no PII in public cache key"). - public + private writes never collide on cache key (defensive). - greeting bucket change re-keys the personalised cache (Brain B parity). - profile change re-keys the personalised cache. - v3 cache prefix used (no v2 writes). Test results: 77/77 in brief-llm; full data suite 6971/6971 (was 6962 pre-Step-7; +9 new public-cache tests). Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md Steps 6 (partial) + 7. Acceptance A5, A6.g, A6.f. * test(digest): backfill A6.h/i/l/m acceptance tests via helper extraction * fix(brief): close two correctness regressions on multi-rule + public surface Two findings from human review of the canonical-synthesis PR: 1. Public-share redaction leaked personalised signals + threads. The new prompt explicitly personalises both `lead` and `signals` ("personalise lead and signals"), but redactForPublic only substituted `lead` — leaving `signals` and `threads` intact. Public renderer's hasSignals gate would emit the signals page whenever `digest.signals.length > 0`, exposing watched-asset / region phrasing to anonymous readers. Same privacy bug class the original PR was meant to close, just on different fields. 2. Multi-rule users got cross-pool lead/storyList mismatch. composeAndStoreBriefForUser picks ONE winning rule for the canonical envelope. The send loop then injected that ONE `briefLead` into every due rule's channel body — even though each rule's storyList came from its own (per-rule) digest pool. Multi-rule users (e.g. `full` + `finance`) ended up with email bodies leading on geopolitics while listing finance stories. Cross-rule editorial mismatch reintroduced after the cross- surface fix. Fix 1 — public signals + threads: - Envelope shape: BriefDigest gains `publicSignals?: string[]` + `publicThreads?: BriefThread[]` (sibling fields to publicLead). Renderer's ALLOWED_DIGEST_KEYS extended; assertBriefEnvelope validates them when present. - generateDigestProsePublic already returned a full prose object (lead + signals + threads) — orchestration now captures all three instead of just `.lead`. Composer splices each into its envelope slot. - redactForPublic substitutes: digest.lead ← publicLead (or empty → omits pull-quote) digest.signals ← publicSignals (or empty → omits signals page) digest.threads ← publicThreads (or category-derived stub via new derivePublicThreadsStub helper — never falls back to the personalised threads) - New tests cover all three substitutions + their fail-safes. Fix 2 — per-rule synthesis in send loop: - Each due rule independently calls runSynthesisWithFallback over ITS OWN pool + ctx. Channel body lead is internally consistent with the storyList (both from the same pool). - Cache absorbs the cost: when this is the winner rule, the synthesis hits the cache row written during the compose pass (same userId/sensitivity/pool/ctx) — no extra LLM call. Only multi-rule users with non-overlapping pools incur additional LLM calls. - magazineUrl still points at the winner's envelope (single brief per user per slot — `(userId, issueSlot)` URL contract). Channel lead vs magazine lead may differ for non-winner rule sends; documented as acceptable trade-off (URL/key shape change to support per-rule magazines is out of scope for this PR). - Parity log refined: adds `winner_match=<bool>` field. The PARITY REGRESSION warning now fires only when winner_match=true AND the channel lead differs from the envelope lead (the actual contract regression). Non-winner sends with legitimately different leads no longer spam the alert. Test results: - tests/brief-magazine-render.test.mjs: 75/75 (+7 new for public signals/threads + validator + private-mode-ignores-public-fields) - Full data suite: 6995/6995 (was 6988; +7 net) - typecheck + typecheck:api: clean Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md Addresses 2 review findings on PR #3396 not anticipated in the 5-round Codex review. * fix(brief): unify compose+send window, fall through filter-rejection Address two residual risks in PR #3396 (single-canonical-brain refactor): Risk 1 — canonical lead synthesized from a fixed 24h pool while the send loop ships stories from `lastSentAt ?? 24h`. For weekly users that meant a 24h-pool lead bolted onto a 7d email body — the same cross-surface divergence the refactor was meant to eliminate, just in a different shape. Twice-daily users hit a 12h-vs-24h variant. Fix: extract the window formula to `digestWindowStartMs(lastSentAt, nowMs, defaultLookbackMs)` in digest-orchestration-helpers.mjs and call it from BOTH the compose path's digestFor closure AND the send loop. The compose path now derives windowStart per-candidate from `cand.lastSentAt`, identical to what the send loop will use for that rule. Removed the now-unused BRIEF_STORY_WINDOW_MS constant. Side-effect: digestFor now receives the full annotated candidate (`cand`) instead of just the rule, so it can reach `cand.lastSentAt`. Backwards-compatible at the helper level — pickWinningCandidateWithPool forwards `cand` instead of `cand.rule`. Cache memo hit rate drops since lastSentAt varies per-rule, but correctness > a few extra Upstash GETs. Risk 2 — pickWinningCandidateWithPool returned the first candidate with a non-empty raw pool as winner. If composeBriefFromDigestStories then dropped every story (URL/headline/shape filters), the caller bailed without trying lower-priority candidates. Pre-PR behaviour was to keep walking. This regressed multi-rule users whose top-priority rule's pool happens to be entirely filter-rejected. Fix: optional `tryCompose(cand, stories)` callback on pickWinningCandidateWithPool. When provided, the helper calls it after the non-empty pool check; falsy return → log filter-rejected and walk to the next candidate; truthy → returns `{winner, stories, composeResult}` so the caller can reuse the result. Without the callback, legacy semantics preserved (existing tests + callers unaffected). Caller composeAndStoreBriefForUser passes a no-synthesis compose call as tryCompose — cheap pure-JS, no I/O. Synthesis only runs once after the winner is locked in, so the perf cost is one extra compose per filter-rejected candidate, no extra LLM round-trips. Tests: - 10 new cases in tests/digest-orchestration-helpers.test.mjs covering: digestFor receiving full candidate; tryCompose fall-through to lower-priority; all-rejected returns null; composeResult forwarded; legacy semantics without tryCompose; digestWindowStartMs lastSentAt-vs-default branches; weekly + twice-daily window parity assertions; epoch-zero ?? guard. - Updated tests/digest-cache-key-sensitivity.test.mjs static-shape regex to match the new `cand.rule.sensitivity` cache-key shape (intent unchanged: cache key MUST include sensitivity). Stacked on PR #3396 — targets feat/brief-two-brain-divergence.
koala73
added a commit
that referenced
this pull request
Apr 25, 2026
…logy doc
Completes Phase 2 of plan 2026-04-25-004 by shipping the 3 component
seeders, registering them in the macro bundle, adding 34 new fixture-
based tests, and writing the full methodology doc with license attribution.
Seeders (per plan §Files (Phase 2)):
scripts/seed-wb-external-debt.mjs — WB IDS short-term external debt as % GNI
(DT.DOD.DSTC.IR.ZS × DT.DOD.DECT.GN.ZS,
mrv=5 + pickLatestPerCountry per memory
`feedback_wb_bulk_mrv1_null_coverage_trap`)
scripts/seed-bis-lbs.mjs — BIS LBS by-parent SDMX integration
(12-dim key with L_POS_TYPE=N per Codex
R3 P1 #1; 16 enumerated parent ISO2 codes
per Codex R4 P1 #2 — no `4F` aggregate;
ISO2 direct, no M49 mapping per CL_BIS_IF_REF_AREA)
scripts/seed-fatf-listing.mjs — FATF entry-page parser with dynamic
publication-URL follow + sanity-check
band gates + monthly cadence + 90d cache
TTL fallback
Bundle registration:
scripts/seed-bundle-macro.mjs — Option A per Codex R1 #5 (less operational
overhead than provisioning a new bundle).
Tests (34 new):
tests/seed-wb-external-debt.test.mjs — combineExternalDebt formula pinning,
IMF Article IV 15% GNI anchor,
conservative-year selection, validate floor
tests/seed-bis-lbs.test.mjs — combineLbsByCounterparty Brazil 2024Q4
ground-truth anchor, 1% GDP threshold
for parentCount, BIS aggregate-code
allow-list (5J/5A skipped), SDMX-JSON
shape parsing, parser-regression guard
tests/seed-fatf-listing.test.mjs — findPublicationLink entry-page anchor
extraction (case-insensitive), country-
name lookup with apostrophe handling
(`People's` → `peoples`), pub-date
inference from URL slug + header,
DPRK-on-call-for-action invariant
Methodology doc:
docs/methodology/financial-system-exposure.md — full construct definition,
per-component formulas + score
shapes + coverage matrices,
fail-closed preflight,
methodology invariants,
sanctions-isolated-jurisdiction
sanity-check anchors (RU/IR/KP/...
< 20), bounded-movement gate
(60%+ |Δ|<3pt), data-source
licensing (BIS terms of use),
alternatives considered (5),
future considerations (Phase 3-5)
Quality:
- npm run typecheck + typecheck:api: clean
- npm run test:data: 7149/7149 pass (was 7115; +34 new seeder tests)
- npm run lint + lint:md + version:check: clean
- Edge function bundle check: clean
Activation runbook (when ready to flip the dim live):
1. Merge this PR
2. Run seed-wb-external-debt + seed-bis-lbs + seed-fatf-listing manually
via `railway run --service seed-bundle-macro -- node scripts/<seeder>.mjs`
3. Verify all 3 seed-meta envelopes published:
redis-cli GET 'seed-meta:economic:wb-external-debt'
redis-cli GET 'seed-meta:economic:bis-lbs'
redis-cli GET 'seed-meta:economic:fatf-listing'
4. Set RESILIENCE_FIN_SYS_EXPOSURE_ENABLED=true in Vercel + Railway env config
5. Flush v14 caches: bulk DEL resilience:score:v14:* + DEL resilience:ranking:v14
6. Run seed-resilience-scores.mjs to bulk-warm v14 with the new dim's signal
7. Cohort audit: snapshot resilience:score:v14:* for all 222 countries;
RU/IR/KP must score < 20 on financialSystemExposure (gate the construct
before stable-rollout)
8. Bounded-movement gate: 60% of countries |Δ|<3pt; outliers > 12pt must
be in the explicitly-predicted RU/IR/KP/CU/VE/BY/LY/MM set
9. Remove FLAG_GATED_DARK_DIMENSIONS allow-list entry in
tests/resilience-release-gate.test.mts in the same commit that flips
the flag
🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v3.0.0
Co-Authored-By: Claude Opus 4.7 (1M context, extended thinking) <[email protected]>
koala73
added a commit
that referenced
this pull request
Apr 25, 2026
…logy doc
Completes Phase 2 of plan 2026-04-25-004 by shipping the 3 component
seeders, registering them in the macro bundle, adding 34 new fixture-
based tests, and writing the full methodology doc with license attribution.
Seeders (per plan §Files (Phase 2)):
scripts/seed-wb-external-debt.mjs — WB IDS short-term external debt as % GNI
(DT.DOD.DSTC.IR.ZS × DT.DOD.DECT.GN.ZS,
mrv=5 + pickLatestPerCountry per memory
`feedback_wb_bulk_mrv1_null_coverage_trap`)
scripts/seed-bis-lbs.mjs — BIS LBS by-parent SDMX integration
(12-dim key with L_POS_TYPE=N per Codex
R3 P1 #1; 16 enumerated parent ISO2 codes
per Codex R4 P1 #2 — no `4F` aggregate;
ISO2 direct, no M49 mapping per CL_BIS_IF_REF_AREA)
scripts/seed-fatf-listing.mjs — FATF entry-page parser with dynamic
publication-URL follow + sanity-check
band gates + monthly cadence + 90d cache
TTL fallback
Bundle registration:
scripts/seed-bundle-macro.mjs — Option A per Codex R1 #5 (less operational
overhead than provisioning a new bundle).
Tests (34 new):
tests/seed-wb-external-debt.test.mjs — combineExternalDebt formula pinning,
IMF Article IV 15% GNI anchor,
conservative-year selection, validate floor
tests/seed-bis-lbs.test.mjs — combineLbsByCounterparty Brazil 2024Q4
ground-truth anchor, 1% GDP threshold
for parentCount, BIS aggregate-code
allow-list (5J/5A skipped), SDMX-JSON
shape parsing, parser-regression guard
tests/seed-fatf-listing.test.mjs — findPublicationLink entry-page anchor
extraction (case-insensitive), country-
name lookup with apostrophe handling
(`People's` → `peoples`), pub-date
inference from URL slug + header,
DPRK-on-call-for-action invariant
Methodology doc:
docs/methodology/financial-system-exposure.md — full construct definition,
per-component formulas + score
shapes + coverage matrices,
fail-closed preflight,
methodology invariants,
sanctions-isolated-jurisdiction
sanity-check anchors (RU/IR/KP/...
< 20), bounded-movement gate
(60%+ |Δ|<3pt), data-source
licensing (BIS terms of use),
alternatives considered (5),
future considerations (Phase 3-5)
Quality:
- npm run typecheck + typecheck:api: clean
- npm run test:data: 7149/7149 pass (was 7115; +34 new seeder tests)
- npm run lint + lint:md + version:check: clean
- Edge function bundle check: clean
Activation runbook (when ready to flip the dim live):
1. Merge this PR
2. Run seed-wb-external-debt + seed-bis-lbs + seed-fatf-listing manually
via `railway run --service seed-bundle-macro -- node scripts/<seeder>.mjs`
3. Verify all 3 seed-meta envelopes published:
redis-cli GET 'seed-meta:economic:wb-external-debt'
redis-cli GET 'seed-meta:economic:bis-lbs'
redis-cli GET 'seed-meta:economic:fatf-listing'
4. Set RESILIENCE_FIN_SYS_EXPOSURE_ENABLED=true in Vercel + Railway env config
5. Flush v14 caches: bulk DEL resilience:score:v14:* + DEL resilience:ranking:v14
6. Run seed-resilience-scores.mjs to bulk-warm v14 with the new dim's signal
7. Cohort audit: snapshot resilience:score:v14:* for all 222 countries;
RU/IR/KP must score < 20 on financialSystemExposure (gate the construct
before stable-rollout)
8. Bounded-movement gate: 60% of countries |Δ|<3pt; outliers > 12pt must
be in the explicitly-predicted RU/IR/KP/CU/VE/BY/LY/MM set
9. Remove FLAG_GATED_DARK_DIMENSIONS allow-list entry in
tests/resilience-release-gate.test.mts in the same commit that flips
the flag
🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v3.0.0
Co-Authored-By: Claude Opus 4.7 (1M context, extended thinking) <[email protected]>
koala73
added a commit
that referenced
this pull request
Apr 25, 2026
…2 — flag-gated dark) (#3407) * feat(resilience): financialSystemExposure dim scaffold (Phase 2 Ship 2 — flag-gated dark) Adds the new `financialSystemExposure` resilience dimension introduced in plan 2026-04-25-004 Phase 2, behind the `RESILIENCE_FIN_SYS_EXPOSURE_ENABLED` env flag. The flag defaults OFF so the dim ships dark — the scorer returns the empty-data shape (score=0, coverage=0, imputationClass=null) until the 3 component seeders (seed-bis-lbs, seed-fatf-listing, seed-wb-external-debt) are populating in production. Rollout pattern matches energy v2 (plan 2026-04-24-001). Components (weights total 1.0 inside the dim): short_term_external_debt_pct_gni 0.35 (WB IDS, lowerBetter, 15-0) bis_lbs_xborder_us_eu_uk_pct_gdp 0.30 (BIS LBS by-parent, U-shape) fatf_listing_status 0.20 (FATF, discrete: black=0, gray=30, compliant=100) financial_center_redundancy 0.15 (BIS LBS by-parent count, higherBetter, 1-10) When the flag flips ON, the scorer preflights all 3 required seed-meta envelopes (the BIS LBS seed serves both Component 2 and Component 4 — no separate Component 4 seeder). Missing envelopes throw `ResilienceConfigurationError(message, missingKeys)` (two-arg form per Codex R3 P1 #2) which `scoreAllDimensions` catches and routes to `imputationClass='source-failure'`. Per-country data gaps inside an otherwise-published envelope are distinct: per-component reads return null, the slot drops out of the weighted blend. Cache prefixes bumped in lockstep with the new dim: resilience:score:v13: → v14: resilience:ranking:v13 → v14 resilience:history:v8: → v9: The scaffold also reweights `tradePolicy` 1.0 → 0.5 in RESILIENCE_DIMENSION_WEIGHTS (the new dim shares the economic-domain weight). This shifts the headline overallScore by ~0.46 points in the flag-off baseline because the half-weighted tradePolicy contributes proportionally less to the coverage-weighted economic-domain mean. When the flag flips on with seeders populated, the new dim's actual signal will rebalance the headline score. What this PR ships: - new `scoreFinancialSystemExposure` + `normalizeBandLowerBetter` U-shape helper - 4 new INDICATOR_REGISTRY entries (BIS-derived tagged non-commercial/enrichment per Codex R1 #8) - api/health.js + api/seed-health.js dual-registry entries for the 3 new seed keys - frontend label map + ResilienceWidget "20 dimensions" copy - methodology doc heading "Financial System Exposure" + indicator table - tests/resilience-financial-system-exposure.test.mts: 13 tests pinning the formula contract + fail-closed preflight + flag-off rollout posture - parity test extension for v14/v9 prefixes - 22-dim count update across release-gate + handlers + indicator-registry tests - flag-gated-dark exemption in release-gate's coverage check What this PR does NOT ship (deliberately deferred): - The 3 component seeders themselves (seed-bis-lbs, seed-fatf-listing, seed-wb-external-debt). These need real-API integration with BIS SDMX, FATF HTML scraping, and WB IDS — best landed as a separate PR with fixture-recorded tests. - Methodology doc `docs/methodology/financial-system-exposure.md` with full alternatives + license attribution. - Bundle registration in scripts/seed-bundle-macro.mjs. - Cohort sanity-check anchor test (RU/IR/KP < 20 on financialSystemExposure) — needs real seeder data. Once those land and seeders populate Redis, flip `RESILIENCE_FIN_SYS_EXPOSURE_ENABLED=true` in Vercel + Railway env config to activate the dim. Stacked on PR #3405 (Phase 1 Ship 1). 🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v3.0.0 Co-Authored-By: Claude Opus 4.7 (1M context, extended thinking) <[email protected]> * feat(resilience): financialSystemExposure component seeders + methodology doc Completes Phase 2 of plan 2026-04-25-004 by shipping the 3 component seeders, registering them in the macro bundle, adding 34 new fixture- based tests, and writing the full methodology doc with license attribution. Seeders (per plan §Files (Phase 2)): scripts/seed-wb-external-debt.mjs — WB IDS short-term external debt as % GNI (DT.DOD.DSTC.IR.ZS × DT.DOD.DECT.GN.ZS, mrv=5 + pickLatestPerCountry per memory `feedback_wb_bulk_mrv1_null_coverage_trap`) scripts/seed-bis-lbs.mjs — BIS LBS by-parent SDMX integration (12-dim key with L_POS_TYPE=N per Codex R3 P1 #1; 16 enumerated parent ISO2 codes per Codex R4 P1 #2 — no `4F` aggregate; ISO2 direct, no M49 mapping per CL_BIS_IF_REF_AREA) scripts/seed-fatf-listing.mjs — FATF entry-page parser with dynamic publication-URL follow + sanity-check band gates + monthly cadence + 90d cache TTL fallback Bundle registration: scripts/seed-bundle-macro.mjs — Option A per Codex R1 #5 (less operational overhead than provisioning a new bundle). Tests (34 new): tests/seed-wb-external-debt.test.mjs — combineExternalDebt formula pinning, IMF Article IV 15% GNI anchor, conservative-year selection, validate floor tests/seed-bis-lbs.test.mjs — combineLbsByCounterparty Brazil 2024Q4 ground-truth anchor, 1% GDP threshold for parentCount, BIS aggregate-code allow-list (5J/5A skipped), SDMX-JSON shape parsing, parser-regression guard tests/seed-fatf-listing.test.mjs — findPublicationLink entry-page anchor extraction (case-insensitive), country- name lookup with apostrophe handling (`People's` → `peoples`), pub-date inference from URL slug + header, DPRK-on-call-for-action invariant Methodology doc: docs/methodology/financial-system-exposure.md — full construct definition, per-component formulas + score shapes + coverage matrices, fail-closed preflight, methodology invariants, sanctions-isolated-jurisdiction sanity-check anchors (RU/IR/KP/... < 20), bounded-movement gate (60%+ |Δ|<3pt), data-source licensing (BIS terms of use), alternatives considered (5), future considerations (Phase 3-5) Quality: - npm run typecheck + typecheck:api: clean - npm run test:data: 7149/7149 pass (was 7115; +34 new seeder tests) - npm run lint + lint:md + version:check: clean - Edge function bundle check: clean Activation runbook (when ready to flip the dim live): 1. Merge this PR 2. Run seed-wb-external-debt + seed-bis-lbs + seed-fatf-listing manually via `railway run --service seed-bundle-macro -- node scripts/<seeder>.mjs` 3. Verify all 3 seed-meta envelopes published: redis-cli GET 'seed-meta:economic:wb-external-debt' redis-cli GET 'seed-meta:economic:bis-lbs' redis-cli GET 'seed-meta:economic:fatf-listing' 4. Set RESILIENCE_FIN_SYS_EXPOSURE_ENABLED=true in Vercel + Railway env config 5. Flush v14 caches: bulk DEL resilience:score:v14:* + DEL resilience:ranking:v14 6. Run seed-resilience-scores.mjs to bulk-warm v14 with the new dim's signal 7. Cohort audit: snapshot resilience:score:v14:* for all 222 countries; RU/IR/KP must score < 20 on financialSystemExposure (gate the construct before stable-rollout) 8. Bounded-movement gate: 60% of countries |Δ|<3pt; outliers > 12pt must be in the explicitly-predicted RU/IR/KP/CU/VE/BY/LY/MM set 9. Remove FLAG_GATED_DARK_DIMENSIONS allow-list entry in tests/resilience-release-gate.test.mts in the same commit that flips the flag 🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v3.0.0 Co-Authored-By: Claude Opus 4.7 (1M context, extended thinking) <[email protected]> * chore(resilience): address self-review hardening (P1 + 4×P2 + 4×P3) P1 — BIS LBS sequential-fetch timeout math: Sequential 16 parents × 60s timeout = 960s worst-case, exceeding the bundle's 600s timeoutMs. SIGTERM mid-flight would have leaked the child-lock + produced a covertly-degraded payload (per memory `bundle-runner-sigkill-leaks-child-lock`). Fix: parallelize parent fetches with concurrency=4 via a bounded-concurrency runner. Caps wall time at ~4 × 60s = 240s on the slow path while staying polite to BIS. P2 — BIS LBS parent-success gate: Previous "all 16 failed" short-circuit was too permissive. If 15 of 16 parent fetches failed, a single-parent payload would pass validate (>100 counterparty floor) and skew Component 4 (financialCenterRedundancy) low for every counterparty until the next successful run. Added MIN_SUCCESSFUL_PARENTS=12 gate; below that, throw → seed-meta unchanged → previous valid payload stays alive under cache TTL. P2 — FATF findPublicationLink prefers highest-year anchor: Previous "first anchor matching label" was vulnerable to FATF page layouts where a sidebar links to historical publications using the same wording. Fix: collect all candidates, sort by year (URL slug or anchor text), return highest. Logs all candidates at WARN when more than one matches so ops can spot drift. P2 — FATF unmatched country-name surfacing: Previous parser silently dropped country names not in shared/country-names.json. If FATF introduced a new spelling ("Mauretania", "Türkiye"), the country fell out of the listing and defaulted to "compliant" (score 100) — materially shifting its financialSystemExposure score under a fresh seed-meta. Fix: extractListedCountries now returns { listed, unmatchedCandidates }. Seeder logs unmatched at WARN; throws if > 2 candidates unmatched (indicates parser drift / new spellings the lookup needs to learn). P2 — WB external-debt yearMismatch metadata: Composition can mix vintages of the two source indicators (different WB IDS lag patterns). Previous output silently used min(year) as the conservative anchor. Fix: emit `yearMismatch: boolean` + `shortTermPctOfTotalDebtYear` + `totalDebtPctOfGniYear` on each country record so the dashboard / scorer / audit can flag countries with cross-year composition. Pinning test asserts min(year) selection + correct flag. P3 — BIS LBS upper-bound corruption guard: Added `latestVal > 1e8` skip in extractClaimsByCounterparty. 1e8 millions = $100T (>half of global GDP), well above any plausible bilateral claim. Drops corrupt SDMX values silently rather than letting them inflate totalXborderPctGdp downstream. P3 — BIS LBS validation floor 100 → 150: BIS LBS counterparty coverage is ~200+ jurisdictions. Floor of 100 would have accepted a payload with massive coverage regression. Tightened to 150. P3 — FATF grey-list floor 8 → 12 + band 8-40 → 12-40: Historical FATF grey-list size has been 15+ since 2020. Floor of 8 was too lenient. Tightened to 12 with comment explaining the historical band. P3 — FATF parse-failure WARN logging: All sanity-check throw paths in seed-fatf-listing now emit a console.warn explaining the failure and noting "previous valid payload remains under cache TTL" before throwing. Plan called for "warn loudly"; the implicit fall-back-to-cache pattern is now diagnostic-friendly. Suggestion — BIS LBS droppedForMissingGdp provenance: Counterparties seen in BIS LBS but dropped because no WB GDP record was available are now collected into a `droppedForMissingGdp` array on the seed payload. Surfaces silent coverage gaps for ops triage without polluting the main `countries` map. Suggestion — methodology doc operational footguns + smoke test: New §"Common operational footguns" section in docs/methodology/financial-system-exposure.md surfacing the BIS LBS 4F-aggregate-rejection lesson + ISO 3166-1 (not M49) clarification + pre-flag-flip smoke test commands. Quality gates: - npm run typecheck + typecheck:api: clean - npm run lint + lint:md + version:check: clean - npm run test:data: 7153/7153 pass (was 7149; +4 hardening tests) 🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v3.0.0 Co-Authored-By: Claude Opus 4.7 (1M context, extended thinking) <[email protected]> * fix(resilience): scoreFinancialSystemExposure preflights UNVERSIONED seed-meta keys P1 reviewer catch: the preflight in scoreFinancialSystemExposure was reading `seed-meta:economic:<key>:v1` while runSeed (scripts/_seed-utils.mjs) writes the freshness record at `seed-meta:${dataKey.replace(/:v\d+$/, '')}` — i.e. with the trailing :v\d+ stripped. Once `RESILIENCE_FIN_SYS_EXPOSURE_ENABLED=true` was flipped, every /api/resilience/* request would have hit the missing- seed-meta path indefinitely, throwing ResilienceConfigurationError and stamping every country's financialSystemExposure as imputationClass='source-failure' even with healthy seeders running. The same unversioned shape is already used by api/health.js + api/seed-health.js + every other in-tree scorer that walks seed-meta keys via _dimension-freshness.ts. Fix: route the preflight through `resolveSeedMetaKey` from _dimension-freshness.ts. That helper already strips the trailing :v\d+ AND applies the SOURCE_KEY_META_OVERRIDES table — the canonical in-tree pattern for "given a registry data-key, return the seed-meta key that runSeed actually writes." Inlining the regex would have re-introduced the same writer/reader drift this helper exists to prevent. Tests: - All formula + preflight tests (which previously mocked the incorrect versioned form) updated to the unversioned key shape so they actually exercise the production read path. - New regression-guard test "preflight reads UNVERSIONED seed-meta keys (matches runSeed write-key shape)" pins the exact key shape the scorer probes. Asserts both presence of the unversioned form AND absence of the versioned form. A future refactor that accidentally re-versions the preflight will fail loudly here instead of silently routing every country to source-failure. Quality gates: - npm run typecheck:api: clean - npm run test:data: 7154/7154 pass (was 7153; +1 contract guard) 🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v3.0.0 Co-Authored-By: Claude Opus 4.7 (1M context, extended thinking) <[email protected]> * fix(resilience): address Greptile P1 + P2 catches on PR #3407 (4 issues) P1 — 30-point scoring cliff at 25% boundary in normalizeBandLowerBetter: Original draft had piecewise-linear segments with mismatched endpoints: - At value=25: sweet spot ended at 100, over-exposed started at 70 → 30-point cliff × 0.30 weight ≈ 9-pt headline swing - At value=5: low-int ended at 70, sweet started at 75 → 5-pt jump Cliffs in piecewise-linear scorers cause ranking instability for countries near band edges (24.9% scores ~99.9, 25.1% scores ~70). Re-anchored adjacent segments to share endpoints — function is now piecewise-CONTINUOUS at 5%, 25%, and 60% transitions: 0%-5%: 60 → 75 (slope +3/pct, was +2) 5%-25%: 75 → 100 (unchanged) 25%-60%: 100 → 30 (slope −2/pct, was 70 → 30 / slope −1.14) 60%+: 30 → 0 (unchanged) New regression test pins continuity at all three boundaries (samples values immediately above/below each transition, asserts |Δ| ≤ 1pt). P2 — readFatfStatus defaults empty listings dict to "compliant": An empty `listings: {}` payload that bypassed the seeder's validate() would silently score every country at 100 (compliant default), masking a parser regression. Added defense-in-depth guard: empty dict → null component score → slot drops out of weighted blend → coverage shrinks visibly rather than the dim looking healthy. Seeder validate already enforces ≥1 black + ≥12 grey so this can't reach production through the normal write path; the guard costs nothing and catches malformed payloads that bypass validation. New regression test pins. P2 — bisLbsXborderPctGdp goalposts mismatch U-shape peak: Registry entry had `goalposts: { worst: 60, best: 15 }` implying a linear lowerBetter scale peaking at 15%. The U-shape function actually peaks at 25% (value=15% scores 87.5, not 100). Updated to `{ worst: 60, best: 25 }` (over-exposed branch, peak anchor) with an explicit comment that goalposts here are documentation-only — the actual scorer uses normalizeBandLowerBetter, not a generic linear normalizer. Tooling reading the registry should consult the scorer helper directly. P2 (resolved differently than originally suggested) — api/bootstrap.js missing 3 new data keys: Greptile flagged that AGENTS.md requires bootstrap hydration for new data sources. Verified the bootstrap-hydration-coverage test enforces this via a `getHydratedData` consumer requirement in src/. The 3 new keys (economic:wb-external-debt:v1, economic:bis-lbs:v1, economic:fatf-listing:v1) are SERVER-ONLY — they feed scoreFinancialSystemExposure inside /api/resilience/* handlers; no client panel consumes them directly. Adding them to bootstrap without a consumer would fail the parity tests. Documented in api/bootstrap.js as a deferred entry: "if a future PR adds a client panel that displays raw BIS LBS / FATF / WB external-debt data, register the keys here AND add the corresponding consumer + cache-keys.ts entries in the same PR." Note on Greptile's P1 #1 (preflight `:v1` suffix mismatch): already resolved in commit d904103 (uses `resolveSeedMetaKey` from _dimension-freshness.ts). Greptile reviewed an earlier commit. Quality gates: - npm run typecheck:api: clean - npm run lint + lint:md + version:check: clean - npm run test:data: 7156/7156 pass (was 7154; +2 regression guards) 🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v3.0.0 Co-Authored-By: Claude Opus 4.7 (1M context, extended thinking) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context, extended thinking) <[email protected]>
koala73
added a commit
that referenced
this pull request
May 10, 2026
* feat(convex): add mcpProTokens table + issue/validate/revoke + http routes (U1)
Non-key Pro MCP identity layer for the Pro-tier MCP access plan. Mirrors
apiKeys.ts shape but stores no key material — the row's _id IS the
bearer identifier (referenced from OAuth code/token records as
mcpTokenId).
- mcpProTokens table: {userId, clientId?, name?, createdAt, lastUsedAt?,
revokedAt?} indexed by_userId; userApiKeys untouched.
- Internal mutations: issueProMcpToken (tier ≥ 1 gate, 5-row cap with
silent oldest-revoke rotation, audit-preserving), validateProMcpToken,
internalRevokeProMcpToken (server-side rollback path for U5 — no Clerk
context needed), touchProMcpTokenLastUsed (debounced 5min).
- Public mutations: revokeProMcpToken + listProMcpTokens (Clerk-auth).
- HTTP internal routes: /api/internal-{issue,validate,revoke}-pro-mcp-token
with x-convex-shared-secret guard. Validate route schedules touch via
ctx.scheduler.runAfter, matching apiKeys pattern (http.ts:839).
28 new tests; full convex suite 245/245.
Plan unit U1: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
* feat(server): pro-mcp-token edge helper — issue/validate/revoke (U2)
Edge-runtime-safe wrappers around U1's Convex internal HTTP routes. No
positive cache on validate (revoke takes effect on next request);
negative-cache only at pro-mcp-token-neg:<tokenId> (60s TTL) for
known-bad bearers.
- issueProMcpTokenForUser: typed ProMcpIssueFailed (pro-required /
invalid-user-id / config / network), 3s timeout.
- validateProMcpToken: neg-cache short-circuit BEFORE Convex hit; fail-soft
on 5xx/timeout (returns null but does NOT write neg-cache sentinel —
prevents transient-blip poisoning of legitimate tokens). Convex's
validate route schedules touchProMcpTokenLastUsed internally; no
edge-side waitUntil needed.
- revokeProMcpToken: returns result object (no throw) so rollback callers
don't have their original error masked. Sets neg-cache sentinel after
successful revoke.
- invalidateProMcpTokenCache: public sentinel writer for U9.
25 tests including a load-bearing integration test for revoke → next
validate short-circuits without Convex round-trip.
Plan unit U2.
* feat(oauth): apex Clerk grant page + signed-grant mint API (U3)
Bridge between the apex Clerk session and the api-subdomain Pro MCP
consent flow. Apex /mcp-grant SPA reads the OAuth nonce + registered
client metadata so users see the real client_name + redirect host
(anti-phishing). On Authorize the page POSTs to mcp-grant-mint, gets
back a fixed redirect URL with a HMAC-signed grant token, and navigates.
- mcp-grant.html + src/mcp-grant-main.ts: Vite multi-page entry (matches
existing settings.html / live-channels.html convention).
- api/internal/mcp-grant-mint.ts: Clerk-auth POST. Re-checks tier ≥ 1,
re-validates redirect_uri allowlist (defense-in-depth), HMAC-signs
{userId, nonce, exp+5min}, SETEX mcp-grant:<nonce>, returns FIXED
redirect URL (https://api.worldmonitor.app/oauth/authorize-pro).
- api/internal/mcp-grant-context.ts: GET companion for the SPA to fetch
the real client_name + redirect_host. Same Pro-tier gate as mint.
- api/_mcp-grant-hmac.ts: shared sign/verify helper. Signature is over
exact post-base64url-decode bytes (key-order/whitespace independent).
U5 imports verifyGrant from this module.
- vercel.json: /mcp-grant → /mcp-grant.html rewrite + no-store header
rule. Catch-all SPA fallback adjusted to exclude the new page.
- api/oauth/register.js: export isAllowedRedirectUri so DCR allowlist is
reused (no parallel impl).
- 35 tests; full deploy-config + edge-functions + mcp + pro-mcp-token
suites green.
Plan unit U3.
* feat(oauth): consent page Pro-CTA-default + 'Use API key instead' (U4)
R1 of the plan: a logged-in Pro user authorising MCP from claude.ai
never sees the API-key input field by default. The consent page now
leads with a brand-green "Sign in with WorldMonitor Pro" CTA pointing
at the apex /mcp-grant page; the existing API-key form is hidden
behind a "Use API key instead" disclosure for Starter+ holders.
- Default state: form display:none; Pro CTA dominant.
- Error state: existing errorMsg path still works — when ke.textContent
is non-empty (or XHR-retry returns invalid_key), inline script auto-
reveals the form so the user doesn't lose context after a bad key.
- Deep-link: /oauth/authorize?...#api-key shows the form on initial
load, so Starter+ users can bookmark.
- Pro CTA href: https://worldmonitor.app/mcp-grant?nonce=<URL-encoded n>.
Apex page reads the OAuth nonce server-side (no client_name forwarded
via URL — already covered by U3's mcp-grant-context).
- nonce shared with U5: the same oauth:nonce:<n> row that U5's
/oauth/authorize-pro will GETDEL. Handler still mints exactly one
nonce per GET — no double-issue.
- XSS-escape preserved on client_name + redirect_host + nonce + errorMsg.
- 18 new tests; existing handler logic untouched.
Plan unit U4.
* feat(oauth): /oauth/authorize-pro bounce-back endpoint (U5)
Receives the apex grant bounce-back, validates the HMAC-signed grant,
atomically consumes both Redis nonces (mcp-grant + oauth:nonce), issues
a non-key Pro identity row in Convex via U2, and writes the OAuth code
with {kind:'pro', userId, mcpTokenId, ...} for U6 to read at exchange
time.
Security ordering (HMAC-first to avoid burning nonces on forged tokens):
1. verifyGrant (U3's _mcp-grant-hmac, no Redis)
2. URL-nonce vs payload-nonce match
3. GETDEL mcp-grant:<n>
4. strict {userId, exp} tuple match vs grant payload (forge defense)
5. GETDEL oauth:nonce:<n>
6. GET oauth:client:<client_id> + redirect_uri allowlist re-check
7. getEntitlements(userId) tier ≥ 1 re-check (lapse defense)
8. issueProMcpTokenForUser (U2)
9. SETEX oauth:code:<code> (kind:'pro', 600s)
10. 302 redirect_uri?code=<code>[&state=...] with Cache-Control:no-store
oauth:code shape (load-bearing for U6):
{kind:'pro', userId, mcpTokenId, client_id, redirect_uri,
code_challenge, scope:'mcp_pro'}
Best-effort revoke when SETEX fails after issueProMcpTokenForUser
succeeds — calls U2's no-throw revokeProMcpToken; logs orphan-row id
on revoke failure but never masks the original 500.
35 tests; vercel rewrite + api-route-exceptions registered as
external-protocol.
Plan unit U5.
* feat(oauth): McpAuthContext discriminated union + Pro token shape (U6)
Token endpoint and bearer resolver learn about Pro-shape tokens without
breaking Starter+ legacy paths.
api/_oauth-token.js:
- New resolveBearerToContext(token) returns a discriminated union:
{kind:'env_key', apiKey} -- legacy WORLDMONITOR_VALID_KEYS resolution
{kind:'pro', userId, mcpTokenId} -- new Pro identity
- resolveApiKeyFromBearer kept as backward-compat wrapper. Returns
apiKey for env_key kind, null for pro kind (so legacy callers can't
mis-handle a Pro bearer).
api/oauth/token.js → api/oauth/token.ts:
- Converted to TypeScript so it can import validateProMcpToken from
server/_shared/pro-mcp-token cleanly. Vercel routes by basename;
/oauth/token rewrite unaffected.
- Default handler wires injected deps via tokenHandler(req, deps) for
testability (mirrors U3/U5 pattern).
- authorization_code branch: dispatches on codeData.kind. Pro path
validates client_id + redirect_uri bind, PKCE-verifies, mints tokens
via storeProTokens (object shape). Legacy path unchanged.
- refresh_token branch: Pro refreshes call validateProMcpToken
(per-request Convex hit, no positive cache); revoked row → invalid_grant.
Cross-user defensive guard: Convex-returned userId must match bearer's.
family_id, mcpTokenId, scope ('mcp_pro') preserved across rotation.
- client_credentials grant untouched.
oauth:token:<uuid> shapes:
- Legacy (preserved): JSON.stringify("<sha256-hex-64>") or "<fingerprint-16>"
- Pro (new): JSON.stringify({kind:'pro', userId, mcpTokenId})
- oauth:refresh:<uuid> Pro shape includes client_id, scope, family_id
for rotation discipline (matches legacy semantics).
26 new tests; sibling suites (oauth-authorize, oauth-authorize-pro, mcp,
pro-mcp-token — 101 tests) regress green. tsc -p tsconfig.api.json clean.
Plan unit U6.
* feat(mcp): Pro identity, atomic INCR-first quota, internal HMAC fetch (U7)
Behavioral heart of the Pro MCP plan. api/mcp.ts now resolves bearers
as McpAuthContext (env_key | pro), runs Pro-specific pre-checks, and
enforces a hard 50/UTC-day cap via INCR-first reservation with
DECR-rollback on cap-exceed and on tool-dispatch failure.
Pre-dispatch chain for Pro context (synchronous, in order):
1. validateProMcpToken(mcpTokenId) — null = 401 revoked
2. defensive userId match — 401 cross-user
3. getEntitlements(userId) tier ≥ 1, mcpAccess: true, validUntil — fail-closed
4. per-minute slidingWindow(60, 60s) keyed pro-user:<userId> — fail-open
5. for tools/call ONLY: pipeline INCR + EXPIRE 172800
newCount > 50 → DECR rollback + -32029 + 429 + Retry-After
INCR Redis transient → -32603 + 503 + Retry-After:5 (hard-cap correctness)
6. dispatch tool; on throw → DECR rollback + -32603
Internal-HMAC tool fetches (replaces X-WorldMonitor-Key for Pro):
payload = ${ts}:${METHOD}:${pathname}:${queryHash}:${bodyHash}:${userId}
queryHash = SHA-256(canonicalQueryString) sorted keys, URL-encoded
bodyHash = SHA-256(bodyBytes) SHA-256("") for empty
sig = HMAC-SHA-256(MCP_INTERNAL_HMAC_SECRET, payload)
headers = X-WM-MCP-Internal: ${ts}.${b64url(sig)}, X-WM-MCP-User-Id
Replay defense: payload binds method+path+queryHash+bodyHash, so a
captured signature for /api/news/v1/list-feed-digest cannot be reused
on /api/intelligence/v1/deduct-situation. ±30s window.
server/_shared/mcp-internal-hmac.ts: single source of truth for sign
helpers. U8 verify imports the same canonicalisation primitives.
server/_shared/pro-mcp-token.ts: dailyCounterKey(userId), 50/172800s
constants exported for U9 to read the SAME Redis key the enforcement
writes.
_execute signatures changed (params, base, context) — option A from
the plan. Cache-only tools unchanged.
waitUntil unused: Convex internal-validate-pro-mcp-token schedules
touchProMcpTokenLastUsed itself (verified at convex/http.ts:1035-1040
from U1's commit 4530bdf).
22 new tests + 23 Starter+ regression tests in tests/mcp.test.mjs;
sibling chain (oauth-token, pro-mcp-token, oauth-authorize-pro,
mcp-grant-mint, mcp-proxy) all green. tsc + biome clean.
Plan unit U7.
* feat(gateway): HMAC-verify internal-MCP requests + sanitised propagation (U8)
Verifier counterpart to U7's signer. Gateway accepts X-WM-MCP-Internal
HMAC headers in lieu of an API key for Pro internal-MCP traffic, then
hands a freshly-constructed Request with sanitised trusted markers to
the downstream handler. isCallerPremium learns to honour those markers
so Pro framework/systemAppend semantics survive in summarize-article,
get-country-intel-brief, deduct-situation.
Gateway flow at the top of createDomainGateway's handler:
1. STRIP inbound x-wm-mcp-internal-verified + x-user-id (anti-injection)
2. If X-WM-MCP-Internal present:
verifyInternalMcpRequest re-canonicalises and HMAC-verifies the
request shape (method+pathname+queryHash+bodyHash+userId), ±30s
window, timing-safe compare. Failure → 401 invalid_internal_mcp_
signature (no fall-through to validateApiKey — present-but-bad is
a deliberate forge attempt).
getEntitlements(userId): tier ≥ 1 + mcpAccess === true (fail-closed).
Construct NEW Request via new Request(url, {method, body, headers})
with x-wm-mcp-internal-verified=<per-process nonce> +
x-user-id=<verified userId>. Skip validateApiKey + IP rate limit.
isCallerPremium adds a NEW first branch: timing-safe compare of
x-wm-mcp-internal-verified against the per-process nonce + non-empty
x-user-id + defensive getEntitlements re-fetch. Catches direct-edge-
function consumers (api/widget-agent.ts, chat-analyst.ts, me/entitlement.ts,
v2/shipping/webhooks/...) that don't run the gateway strip step.
DEFENSE-IN-DEPTH BEYOND PLAN: trusted marker is a per-process random
16-byte nonce, NOT the literal '1'. Sweep found direct edge functions
calling isCallerPremium without gateway-strip protection — a constant
marker would be spoofable from outside. The nonce is born once at
process startup, only the gateway knows it, comparison is timing-safe.
verifyInternalMcpRequest lives in server/_shared/mcp-internal-hmac.ts
next to U7's sign helpers — single source of truth for canonical form
+ payload + ±30s window. Drift between sign and verify is structurally
impossible.
26 new tests + 100 sibling regression tests (gateway-cdn-origin-policy,
premium-stock-gateway, premium-fetch, pro-mcp-token, mcp). tsc clean.
Plan unit U8.
* feat(catalog): mcpAccess feature flag + env vars + Pro-MCP docs (U10)
Catalog, schema, type, env, and docs scaffolding to make the Pro MCP
flow deployable end-to-end. U9 unblocks on this — settings UI gates on
hasFeature('mcpAccess') (distinct from existing apiAccess paywall).
- convex/config/productCatalog.ts: PlanFeatures.mcpAccess on every tier.
Free=false. Pro_monthly/Pro_annual=true. API_starter / API_starter_annual
/ API_business=true. Enterprise=true. Pro plan marketing copy mentions
"MCP access for Claude Desktop & other AI clients (50 calls/day)".
- convex/schema.ts + convex/payments/cacheActions.ts: mcpAccess optional
field on the entitlements validator; legacy rows pass.
- server/_shared/entitlement-check.ts: CachedEntitlements.features
mcpAccess?: boolean (consumed by U7 + U8).
- src/services/entitlements.ts: EntitlementState.features.mcpAccess?:
boolean — unblocks hasFeature('mcpAccess') for U9's settings tab gate.
- .env.example: new "Pro MCP" section with MCP_PRO_GRANT_HMAC_SECRET
(apex grant bridge, U3/U5) and MCP_INTERNAL_HMAC_SECRET (gateway
service-auth, U7/U8). Both 32-byte base64; both required.
- docs/mcp-server.mdx: "Pro sign-in flow" section + 50/day quota
callout + "Connected MCP clients" management pointer.
Bundled fixups for U7/U8 strict-mode TS errors:
- mcp-internal-hmac.ts: bufferToBase64Url uses for-of (avoids
noUncheckedIndexedAccess error on bytes[i]).
- gateway.ts: drop unused INTERNAL_MCP_USER_ID_HEADER import.
tsc -p tsconfig.api.json + tsc -p tsconfig.json clean.
9 entitlements tests + 71 gateway-internal-mcp + mcp tests pass.
Plan unit U10.
* feat(settings): Connected MCP clients tab + quota endpoint + revoke (U9)
User-facing surface that makes the Pro MCP plan visible. Settings UI
gets a new "Connected MCP clients" tab gated on hasFeature('mcpAccess')
(distinct from the existing apiAccess-gated 'API Keys' tab — Pro users
without apiAccess see only the new tab).
Endpoints:
- GET /api/user/mcp-quota → {used, limit:50, resetsAt}. Reads the
SAME Redis key (dailyCounterKey from U2) that U7 enforcement writes.
Single source of truth.
- POST /api/user/mcp-revoke {tokenId} → forwards Clerk-derived userId
(NOT client-supplied) to Convex internal-revoke-pro-mcp-token, then
calls invalidateProMcpTokenCache so any in-flight bearer with this
tokenId 401s within 60s. 404 on NOT_FOUND, 409 on ALREADY_REVOKED.
Tenancy enforced inside Convex (row.userId === userId).
UI (in src/components/UnifiedSettings.ts, no child component split —
matches existing API Keys tab pattern):
- Tab gated on hasFeature('mcpAccess') so Pro users see it without
needing apiAccess.
- Quota header auto-refreshes every 30s; interval cleared on tab-
switch / close / destroy.
- Each row: name, createdAt, lastUsedAt (relative), revokedAt
(struck-through + badge); active rows show Revoke button with
confirm() dialog (matches existing pattern).
- Empty state: click-to-copy https://api.worldmonitor.app/mcp.
Frontend service in src/services/mcp-clients.ts: listMcpClients (Convex
query), revokeMcpClient (POST /api/user/mcp-revoke), fetchMcpQuota.
11 + 14 = 25 new tests; api-route-exceptions registered as
internal-helper.
Plan unit U9.
---
PLAN COMPLETE — 10/10 units shipped. Follow-up polish noted in U9
report: CSS rules for .mcp-clients-* classes (suggest copy from
.api-keys-* rules); empty-state URL is hardcoded to
api.worldmonitor.app/mcp.
* style(settings): mcp-clients CSS rules mirroring api-keys (U9 polish)
* fix(pro-mcp): apply Tier-2 code-review residuals — security + correctness
Review pass after U10 (3 reviewers: code-reviewer, ce-security-reviewer,
ce-adversarial-reviewer). Findings BLOCKING + HIGH + MEDIUM + LOW
applied per user direction.
BLOCKING (gateway):
- Add validUntil check to gateway's HMAC-verify entitlement re-check.
Defense-in-depth against Convex-fallback paths returning a stale row
past its expiry.
HIGH — adv-001: cross-user OAuth nonce hijack:
- mcp-grant-mint now claims oauth nonces atomically via Upstash SET NX
semantics. If the nonce is already claimed by a DIFFERENT userId,
return 403 NONCE_CLAIMED_BY_OTHER_USER. Same userId multi-tab retry
is idempotent: re-sign the grant using the existing claim's exp so
authorize-pro's strict tuple equality still matches.
- mcp-grant-context performs the same cross-user check so the apex SPA
refuses to render consent context for a hijacked nonce.
MEDIUM — adv-008: refresh-token loss during Convex transient:
- validateProMcpToken returns a discriminated union {ok:'valid',userId}
| {ok:'revoked'} | {ok:'transient'}. Negative-cache writes only on
'revoked'. validateProMcpTokenOrNull wrapper preserves backward
compat for callers that don't need the distinction (per-request MCP
edge stays fail-closed).
- token.ts refresh path: on 'transient', best-effort restore the
consumed refresh token to Redis with the original TTL and return
503 server_error + Retry-After:5. Client retries; if Convex recovers,
refresh succeeds. No more permanent session loss on a Convex blip.
MEDIUM — adv-002: counter overshoot lockout:
- mcp.ts: on cap-exceeded after DECR rollback, if newCount is still
far above PRO_DAILY_QUOTA_LIMIT (overshoot from prior transient),
pipelined INCR+DECR probe + bounded DECR-sweep to converge the
counter back near the limit. Prevents one Redis hiccup from locking
out a paying Pro user for the rest of the UTC day.
MEDIUM — code-reviewer #4: 5-row cap race in Convex:
- mcpProTokens.issueProMcpToken now revokes ALL active rows beyond
MAX-1 (sorted by createdAt) on each issue, so concurrent inserts
that briefly produce 6 active rows converge back to 5 on the next
call.
LOW:
- adv-003: executeTool throws cache_all_null when every cache read
returns null AND there were keys configured — triggers DECR rollback
in dispatchToolsCall instead of silently burning quota on a degenerate
empty result.
- code-reviewer #10: gateway strips X-WM-MCP-Internal + X-WM-MCP-User-Id
from the trusted Request before forwarding to handler (no info leak).
- adv-004: 256 KB body cap (Content-Length + post-buffer count) on
both gateway strip and HMAC-verify paths — bounds memory amplification.
- adv-006: dailyCounterKey now env-prefixes (preview deploys can't
collide with production traffic on the same Upstash instance).
Reader (mcp-quota.ts) and writer (mcp.ts) share byte-identical keys
via the helper.
- adv-007 / code-reviewer #5: signInternalMcpRequest throws on
Blob/FormData/ReadableStream/plain-object bodies instead of silently
JSON.stringify'ing them (which would produce a hash that can't match
the wire bytes).
- code-reviewer #9: timestamp regex tightened to ^[0-9]{1,15}$ so
future ms-precision timestamps don't silently truncate through Number().
- code-reviewer #8: MCP_INTERNAL_HMAC_SECRET preflight in runProPreChecks
surfaces config errors at auth-resolution rather than mid-tool-fetch.
NITPICK:
- code-reviewer #6: rewritten readNegCache comment.
- code-reviewer #7: malformed-body error now reports reason
'malformed_request' instead of misleading 'auth_401'. New
RequestReason value added to usage.ts enum.
30 new tests + ~14 adapted; full convex suite 247/247, edge/server
suites 254/254. tsc clean across both tsconfig.json and tsconfig.api.json.
Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
* chore(pro-marketing): regenerate /pro bundle for U10 catalog copy
U10's productCatalog.ts added an mcpAccess feature flag and updated the
Pro plan description copy to "MCP access for Claude Desktop & other AI
clients (50 calls/day)". scripts/generate-product-config.mjs cascades
that copy into pro-test/src/generated/tiers.json, which the /pro
marketing app builds into public/pro/assets/.
Vercel does NOT rebuild pro-test on deploy — public/pro/ ships whatever
is committed. Pre-push hook auto-ran the build and produced a new bundle
hash; this commit lands the regenerated artifacts so deploy picks them up.
Plan unit U10 (follow-on artifact).
* fix(pro-mcp): apply external review round-2 findings (P1+P2+P3)
Round-2 review on PR #3646 surfaced 5 must-fix issues:
P1 — vercel.json `(?:...)` rejected by Vercel CI:
- Replace `mcp-grant(?:\\.html)?` with explicit `mcp-grant\\.html|mcp-grant`
inside the SPA catch-all negative-lookahead (lines 18 + 134).
- Split the headers `/mcp-grant(?:\\.html)?` source rule into two explicit
entries — Vercel's path-to-regexp `source` field doesn't accept `(?:...)`
with `?` quantifier as a standalone source.
- Update tests/deploy-config.test.mjs expectations to match.
P1 — api/api-route-exceptions.json points at deleted file:
- Update the entry path from `api/oauth/token.js` (deleted in U6) to
`api/oauth/token.ts`. `node scripts/enforce-sebuf-api-contract.mjs`
was failing pre-push CI.
P1 — mcpAccess migration gap on existing entitlement rows:
- convex/entitlements.ts now read-time merges `getFeaturesForPlan(planKey)`
defaults under stored features. Pre-U10 rows lacking `mcpAccess` see the
catalog default surfaced immediately, with NO wait for the next Dodo
webhook to rewrite the row. Stored features still win on conflict
(preserves per-user overrides). Mirrored explicitly in
convex/mcpProTokens.ts:55 (issueProMcpToken's direct ctx.db.query path
computes the same merge inline since it bypasses the query handler).
P2 — grant/issue path missing mcpAccess gate (let tier-1-without-mcpAccess
users complete OAuth then fail every tools/call at the gateway):
- mcp-grant-context.ts:95, mcp-grant-mint.ts:236, authorize-pro.ts:356,
convex/mcpProTokens.ts:55 now ALL gate on `tier ≥ 1 && mcpAccess === true`,
matching the downstream MCP-edge runProPreChecks check. Widens the
`getEntitlements` deps return type at all three handlers.
P3 — inline theme script blocked by global CSP:
- mcp-grant.html:34 inline `<script>` is removed (global CSP at vercel.json:92
is hash-allowlist-based and we don't allowlist this page's hashes).
- Theme application moved into the bundled module at src/mcp-grant-main.ts
(top of file, runs on module evaluation). Brief default-theme flash on
light-preference users is acceptable for a transient consent UI.
10 new tests covering: mcpAccess: false at each of the 4 gates,
mcpAccess: undefined (legacy row) at each of the 4 gates, and the
read-time catalog merge in both directions (legacy gets default,
override preserved).
Full gate post-fix:
- Convex 249/249 (+2 new merge tests).
- Edge/server 280/280 (+10 new gate tests).
- tsc both configs clean.
- sebuf contract clean (was failing before P1.2).
- Sentry coverage clean.
- deploy-config tests assert new explicit-source rules.
Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
* fix(entitlement-cache): treat pre-U10 cache entries lacking mcpAccess as stale
Reviewer round-2 P2 (cache layer): _getEntitlementsImpl returned hot
Redis cache entries as-is, bypassing the read-time catalog merge that
convex/entitlements.ts:50 applies on the Convex read path. Paying users
with cache entries written before plan 2026-05-10-001 U10 deployed see
mcpAccess !== true at the grant/MCP gates and are blocked for up to
the 15-min cache TTL.
Cache predicate now also requires `typeof features.mcpAccess === 'boolean'`.
Legacy entries (mcpAccess: undefined) fall through to Convex, which
returns the merged shape and the cache is rewritten with the post-U10
layout. Self-healing, bounded to one extra Convex round-trip per
affected user during the migration window.
Targeted choice over alternatives:
- Importing convex/config/productCatalog into server/_shared to apply
the merge at the cache layer would pull non-edge-safe deps. Rejected.
- Treating ALL cached entries as stale would defeat the cache layer.
Rejected.
- typeof check on the new field is the minimum delta that fixes the
migration window without restructuring.
Test fixture in server/__tests__/entitlement-check.test.ts updated:
makeEntitlements now includes mcpAccess (tier ≥ 1 → true). Two new
focused tests:
1. Legacy cache entry without mcpAccess → cache rejected → Convex
round-trip → result returned (verifies fetch called once).
2. Post-U10 cache entry WITH mcpAccess → cache honored → no Convex
round-trip (verifies fetch called zero times).
Convex 251/251 (+2 new cache tests).
Plan: docs/plans/2026-05-10-001-feat-pro-mcp-clerk-auth-quota-plan.md
This was referenced May 17, 2026
koala73
added a commit
that referenced
this pull request
May 26, 2026
…ns + queries + service + UI (#3621) * feat(country-codes): add toIso2 normalizer for alpha-2/3/name input (U1) Foundation for the followed-countries watchlist primitive. Normalizes any country input form (ISO-3166 alpha-2, alpha-3, lowercase, common country names) to canonical alpha-2 uppercase. Returns null on unrecognized input. Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U1 * feat(convex): add followedCountries + aggregate counter tables (U12) Two new Convex tables for the followed-countries watchlist primitive: - followedCountries: { userId, country, addedAt } indexed by_user, by_country, by_user_country. Per-country reverse lookup via by_country enables future fan-out (digest, breaking-news relay). - followedCountriesCounts: { country, count, updatedAt } indexed by_country. Aggregate counter maintained atomically by U13 mutations so public countFollowers query is O(1) per call rather than O(n) on by_country.collect(). Constants in convex/constants.ts: FREE_TIER_FOLLOW_LIMIT=3 (server- authoritative cap), MAX_MERGE_INPUT=100 (anti-abuse ceiling), COUNTRY_COUNT_PRIVACY_FLOOR=5 (returned-as-zero threshold for public counts). No migration; both tables empty on creation. Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U12 * feat(convex): add followedCountries mutations + ISO-2 registry validator (U13) Three server-authoritative mutations on the new followedCountries table, with atomic counter maintenance for the aggregate followedCountriesCounts table: - followCountry({country}): auth + ISO-2 registry validate + idempotent on (userId, country) + free-tier cap (server-enforced via ConvexError({kind:'FREE_CAP'})) + atomic counter +1. - unfollowCountry({country}): auth + validate + idempotent + atomic counter -1 (max(0, ...) defensive). - mergeAnonymousLocal({countries}): auth + MAX_MERGE_INPUT ceiling (anti-abuse) + ISO-2 registry filter + first-seen dedupe + bounded accept for free users (cap-fitting; over-cap → droppedDueToCap[]) + atomic counter +N. ISO-2 registry validator at convex/lib/iso2.ts mirrors the canonical alpha-2 set from src/utils/country-codes.ts. Both registries must stay in lockstep (documented inline). All errors typed ConvexError({kind, ...}) with object data per the convex-error-string-data-strips-errordata-on-wire memory. 32 new tests covering: auth/validation, free-tier cap (under, at, exceeded), idempotency for both follow + unfollow, counter correctness across all paths (never goes negative), mergeAnonymous with grandfather rejection / partial accept / oversized input / duplicate inputs / mixed valid+invalid. Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U13 * feat(convex): followedCountries queries + relay endpoint (U14) Three queries + one relay HTTP action over the followedCountries + followedCountriesCounts tables: - listFollowed = query({}): auth'd reactive query for current user; returns string[] sorted by addedAt asc; [] when no auth identity. Drives the client-side reactive subscription (U2/U3). - countFollowers = query({country}): public no-auth query backed by the aggregate counter table — O(1) per call, not O(n) on by_country.collect(). Privacy floor (COUNTRY_COUNT_PRIVACY_FLOOR=5) returns 0 below threshold to limit follower-set inference. Drives future 'X people watching' social-proof UI. - listFollowersPage = internalQuery({country, cursor?, limit}): internal-only paginated cursor on by_country index, limit clamped [1, 500]. NEVER exposed publicly (declared as internalQuery, NOT query — the typecheck-level privacy boundary). Drives future per-country fan-out (digest, breaking-news relay). - internalListFollowedForUser = internalQuery({userId}): internal helper used by the relay endpoint (which has no Clerk identity). POST /relay/followed-countries HTTP action mirrors the existing /relay/user-preferences pattern: shared-secret auth via timingSafeEqualStrings, body {userId}, returns {countries: string[]}. Used by PR C's brief composer to read followed-countries server-side. 29 new tests covering: per-user reads, sort order, no-auth empty, counter-table-backed counts, privacy floor edges (4 vs 5), cursor pagination across multi-page result, limit clamp [1,500], @ts-expect-error privacy assertion that listFollowersPage is NOT on api.* (only internal.*), relay 200/400/401 paths. Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U14 * feat(followed-countries): client service — anonymous mode + entitlement gating (U2) Single client-side owner of watchlist semantics for the followed- countries primitive. U2 ships the anonymous (localStorage) path fully working; signed-in mode plumbing is stubbed with explicit TODO(U3) markers for the next unit to fill in. Public API: - getFollowed(): string[] - isFollowed(code: string): boolean - addCountry(input): Promise<FollowMutationResult> - removeCountry(input): Promise<FollowMutationResult> - subscribe(handler): unsubscribe - serviceEntitlementState(): 'pro' | 'free' | 'loading' - WM_FOLLOWED_COUNTRIES_CHANGED custom event Discriminated-union return (memory: discriminated-union-over-sentinel- boolean): { ok: true } | { ok: false, reason: 'DISABLED' | 'INVALID_INPUT' | 'FREE_CAP' | 'ENTITLEMENT_LOADING' | 'HANDOFF_PENDING' | 'STORAGE_FULL' }. Never throws. Anonymous-vs-loading distinction (Codex deepening round-1 P1): serviceEntitlementState() returns 'free' when getCurrentClerkUser() is null, regardless of entitlement state — anonymous users never block on entitlement loading. Only signed-in users with null entitlement state enter 'loading'. Storage: localStorage 'wm-followed-countries-v1' = JSON.stringify( { countries: string[] }). NOT enrolled in CLOUD_SYNC_KEYS — the dedicated Convex table replaces that path for this feature. Feature flag VITE_FOLLOW_COUNTRIES_ENABLED gates all mutations at the service layer (refusal at top of addCountry/removeCountry). Default ON; only '0' disables. 25 tests covering: happy paths, normalization (alpha-3 → alpha-2), idempotency, FREE_CAP cap enforcement, PRO unlimited, ENTITLEMENT_ LOADING (signed-in only), anonymous-never-loads, feature-flag DISABLED, corrupt/wrong-shape localStorage, STORAGE_FULL on quota throw. Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U2 * feat(followed-countries): sign-in handoff + Convex bridge (U3) Wires the signed-in mode of the followed-countries service: - Auth-state listener installed once at app boot. On every Clerk user transition, increments _handoffGeneration and captures userIdAtStart for any in-flight handoff. Post-await callbacks verify both BEFORE clearing localStorage / subscribing — resolves the in-flight auth race (Codex deepening round-1 P1). - Sign-in handoff orchestrator reads localStorage, calls api.followedCountries.mergeAnonymousLocal, clears localStorage on success, subscribes to listFollowed reactive query. - handoffPending UX: addCountry/removeCountry return HANDOFF_PENDING so FollowButton can show a syncing tooltip; getFollowed unions localStorage with the user-scoped subscription snapshot for stable display, BUT only if snapshot.userId matches current Clerk userId (Codex deepening round-2 P1 — no cross-user leak). - Sign-out / user-A → user-B: increments _handoffGeneration, unsubscribes, CLEARS _lastKnownSubscriptionSnapshot = null, resets _handoffState. localStorage retained for next anon session. - Network failure: _handoffState = 'failed', visibilitychange retry. Idempotency means safe to re-run mergeAnonymousLocal. - Convex error → reason mapping via err.data.kind (memory: convex-error-string-data-strips-errordata-on-wire). FREE_CAP preserves currentCount + limit. - Cap-drop event: when mergeAnonymousLocal returns droppedDueToCap[], dispatch WM_FOLLOWED_COUNTRIES_CAP_DROP for the upgrade-CTA toast (consumed by U4 FollowButton). 24 new sign-in handoff tests covering: empty/corrupt localStorage skip + clean, free-tier bounded accept with cap-drop event, network failure → visibilitychange retry, in-flight auth-race sign-out (gen guard drops result), in-flight user-swap (userIdAtStart guard drops result), HANDOFF_PENDING blocks writes, getFollowed user-scoped union, sign-out clears snapshot, sign-in→sign-out→different-user flow, reactive snapshot updates. Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U3 * feat(follow-button): reusable star button helper for 3 surfaces (U4) Single mountable factory used by CountryDeepDivePanel, CountryIntelModal, and CIIPanel rows in U5. Owns: - Visual states: outlined-star (unfollowed), filled-star (followed), spinner (entitlement loading), hidden (feature flag off). At-cap state shows 'Upgrade to follow more' tooltip pre-click. - Click handler: addCountry/removeCountry. FREE_CAP triggers the existing upgrade flow (mirroring notifications-settings.ts lazy- import pattern: openSignIn for anon, startCheckout for signed-in, fallback to /pro#pricing). HANDOFF_PENDING / DISABLED / loading → defensive no-op. - Reactivity: subscribes to WM_FOLLOWED_COUNTRIES_CHANGED and onEntitlementChange; teardown unsubscribes both. Idempotent teardown. - Anonymous-vs-loading distinction (Codex round-2 P1): driven by serviceEntitlementState() helper, NOT raw getEntitlementState(). Anonymous users render interactive (state a/b), only signed-in- awaiting-snapshot enters spinner state. Adds isFollowFeatureEnabled() exported helper to the service so the button gates on the same source of truth as the service. 18 tests covering visual states, anonymous click flow, entitlement- loading window with PRO/FREE resolution, subscription + teardown. Cap-drop toast (WM_FOLLOWED_COUNTRIES_CAP_DROP from U3) is NOT wired here — left as TODO for App-level toast service. CSS is semantic class names only; styling lands in PR B. Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U4 * feat(panels): mount FollowButton on Deep Dive, Intel Modal, CII rows (U5) Surfaces the followed-countries primitive on the three PR-A entry points. No other behavior changes (pin-to-top, filter chips, brief weighting are PR B / PR C). CountryDeepDivePanel: - FollowButton (size: md) inserted in header next to title - Teardown wired into resetPanelContent() + hide(); idempotent CountryIntelModal: - FollowButton (size: md) in header between country name and level - Mount on show(); teardown on showLoading()/hide(); idempotent CIIPanel: - FollowButton (size: sm) as first child of every .cii-country row - Map<countryCode, teardown> tracks per-row mounts; cleared on every wholesale rebuild + on destroy() to prevent listener leaks - Click stopPropagation so star toggle does NOT also trigger row-level onCountryClick (mirrors the existing cii-share-btn pattern) Semantic class names only (.wm-follow-btn + per-host wrappers); CSS lands in PR B's UX polish. Verification: npm run typecheck + typecheck:api clean; full test:data suite still green (7898/7898). Plan: docs/plans/2026-05-02-001-feat-followed-countries-watchlist-primitive-plan.md U5 * fix(followed-countries): convex server-side hardening pass (Phase 1) P3 #21: followCountry now reads entitlement tier BEFORE collecting all user rows; PRO callers skip the O(N) `.collect()` of followedCountries since they have no cap to check. Free users are unchanged. P2 #12: countFollowers privacy-floor doc/code alignment — comment now matches the `<` comparator (1-4 followers → 0; 5+ → exact count). P2 #19: /relay/followed-countries userId validation tightened to mirror /relay/user-preferences rigor — non-empty string with bounded length (<=256 chars) instead of just truthy. Mitigates oversized / non-string abuse vectors. P2 #13: ISO-2 dual-registry parity test upgraded from size-only (`.size === 239`) to set-equality. Catches drift where one side has, e.g., 'XK' and the other has 'EU' with the same total count. `ISO2_TO_ISO3` is now exported from `src/utils/country-codes.ts`. Tests: 278 → 283 (added 1 set-equality, 1 PRO-skip-collect, 3 relay userId validation tests). Plan/Review reference: ce-code-review run 20260502-195816-dae403d7 * fix(followed-countries): service-core hardening pass (Phase 2) P1 #3: _runHandoff catch now uses _extractConvexErrorKind. Permanent ConvexError kinds (INPUT_TOO_LARGE, EMPTY_INPUT, UNAUTHENTICATED) skip the visibilitychange retry path: clear localStorage, transition to new 'failed-permanent' state, install the reactive subscription so signed-in reads still work. Transient errors (network / undefined kind) still retry. P1 #4: max-retry counter (5) + exponential backoff (1, 2, 4, 8, 16 seconds) gate the visibilitychange retry path. After exhaustion the state flips to 'failed-permanent' and no further retries are scheduled. _clearFailedHandoffForTests() exposed as the test recovery hook. Production has no equivalent today; sign-out / sign-in starts a fresh generation. Test seam _setHandoffBackoffForTests collapses the backoff schedule so tests don't have to wait seconds. P1 #5: signed-in addCountry/removeCountry now return HANDOFF_PENDING when the convex client is null instead of falling back to localStorage. Stale partial-writes that never reconcile with the authoritative table are no longer possible in signed-in mode. _setDepsForTests gains a 'force-null' literal so test injection can return null without falling through to the production importer. P1 #6: dropped the `if (existing.includes(code)) return {ok:true}` short-circuit on the SIGNED-IN branch of addCountry/removeCountry. The Convex mutation is itself idempotent and authoritative; the client-side snapshot is eventually consistent and could lie (e.g., another tab just unfollowed). Anonymous-mode short-circuit retained because localStorage IS the source of truth there. P1 #8: replaced unknown-typed ConvexClientLike/ConvexApiLike with FunctionReference<...> generics from convex/server. Mutation arg/result shapes (e.g., {country: code} vs {countries: code}) are now checked at the call site, eliminating the entire typo class. P1 #9: imports MergeAnonymousLocalResult from convex/followedCountries instead of hand-rolling an inline subset. P1 #10: cross-tab `storage` event listener installed alongside the auth-state listener. Filters on key === FOLLOWED_COUNTRIES_STORAGE_KEY and re-dispatches as WM_FOLLOWED_COUNTRIES_CHANGED so FollowButtons in other tabs re-render after a Tab-A mutation. P1 #11: signed-in addCountry/removeCountry capture {userIdAtStart, genAtStart} BEFORE the await, then call _authStillMatches() after the await. A sign-out / user-swap mid-mutation surfaces as HANDOFF_PENDING instead of letting user-A's success "land" while we're already user-B. P2 #20: empty-handoff path defers dispatchChanged until the first reactive snapshot lands. Tracks _initialSnapshotReceived; getFollowed() falls back to localStorage during the gap between 'complete' being set and the first onUpdate callback firing. Avoids a brief flash of empty-list rendering during the subscription warm-up window. Tests: data 7898 → 7911 (added 13 — INPUT_TOO_LARGE/EMPTY_INPUT/ UNAUTHENTICATED permanent-kind handling, plain-error transient guard, max-retry exhaustion + recovery hook, client-null HANDOFF_PENDING for both add and remove, P1 #6 stale-snapshot non-short-circuit, cross-tab storage event re-dispatch x2, post-await auth re-check, empty-handoff deferred dispatch). Convex 283/283 unchanged. Plan/Review reference: ce-code-review run 20260502-195816-dae403d7 * fix(follow-button): UI hardening — exhaustive switch + inFlight guard (Phase 3) P2 #16: added `assertNever(result)` default branch to the FollowButton onClick switch on `FollowMutationResult.reason`. When every variant is handled by a `case`, `result` narrows to `never` at the default; adding a new reason to `FollowMutationResult` will widen the residual type and produce a TS2345 ('not assignable to never') at typecheck time. Catches the future-variant bug class at compile time, with a runtime fallback for malformed test fakes. P2 #17: introduced an `inFlight` boolean closed over by the click handler. When a mutation is already pending, additional clicks are dropped silently (no duplicate addCountry/removeCountry fires). Cleared in finally{} so a thrown service-layer error doesn't latch the button. Without this, a rapid double-click on an unfollowed button produced TWO follow mutations — the service is idempotent on (user, country) but the second add was wasted network + counter-increment work and a toggle pattern (click on, click off) could land in the unintended state. Tests: data 7911 → 7913 (added inFlight rapid-double-click suppression test + assertNever runtime-guard sanity test). Plan/Review reference: ce-code-review run 20260502-195816-dae403d7 * chore(env): document VITE_FOLLOW_COUNTRIES_ENABLED in .env.example (Phase 4) P2 #18: adds the missing .env.example entry for the followed-countries feature flag. Mirrors the format of sibling flags (VITE_CLOUD_PREFS_ENABLED) and clarifies the default-on-unless-'0' semantics enforced by `isFeatureFlagEnabled()` in src/services/followed-countries.ts. Plan/Review reference: ce-code-review run 20260502-195816-dae403d7 * fix(followed-countries): per-user serialization doc — close TOCTOU cap-bypass (P0) Codex round-3 review run 20260502-195816-dae403d7 (adv-001 / adv-002) flagged a P0 cap-bypass on `followCountry` and `mergeAnonymousLocal`. Convex per-document OCC tracks reads at the DOCUMENT level, not at the index-range level — so two parallel `followCountry` mutations from the same user can both read empty/ under-cap from `followedCountries` (an index range), both pass the cap check, and both insert. Cap bypass + potential duplicate (userId, country) rows. The same shape applies to `mergeAnonymousLocal` from N tabs at sign-in. Mitigation: a per-user serialization document (new table `followedCountriesUserMeta`) that every mutation reads AND writes. Convex's real OCC then forces concurrent same-user mutations to serialize on this row: the loser of the race retries, re-reads the post-winner state (which now contains the winner's `(userId, country)` row + bumped count), and either passes correctly (still under cap), throws `FREE_CAP`, or returns idempotent. The denormalized `count` field also makes the cap check O(1) — happy side effect that closes P3 #21 (`.collect()` for cap purposes is gone). Schema: new table `followedCountriesUserMeta` keyed by userId, with a denormalized `count` and `updatedAt`. Convex auto-deploys schema before handlers in the same push, so single-PR shipping is safe. Mutations: - `followCountry`: read meta first, use `count` for cap check (free tier only), patch/insert meta at the END after row insert + counter +1. Idempotent (already-followed) path skips meta write — no observable change, no race to lose. - `unfollowCountry`: read meta first, decrement to `Math.max(0, count - 1)` at the end. Idempotent (no-row) path skips meta write. - `mergeAnonymousLocal`: read meta for the cap denominator, `existingRows` still required for the dedup set. Patch meta with `existingCount + accepted.length` at the end. Skip meta write when `accepted.length === 0`. Tests (+9, total 283 → 292): - Concurrent same-user same-country `Promise.all` → exactly 1 row + 1 idempotent response. - Concurrent same-user cap-boundary (2 seeded + 2 attempts on cap=3) → exactly 1 fulfilled, 1 rejected with FREE_CAP, final ≤ cap. - Concurrent mixed follow/unfollow → consistent end state, parity invariant. - Concurrent `mergeAnonymousLocal` from 5 tabs (free user) → final ≤ cap, no duplicate (userId, country) rows. - Concurrent `mergeAnonymousLocal` from 5 tabs (PRO user) → exactly the deduped union, all per-country counters at 1. - Meta-count parity invariant after mixed mutation sequence. - Idempotent paths (followCountry on existing, unfollowCountry on absent) don't bump/decrement meta count. - FREE_CAP throw rolls back all writes (transaction atomicity). Concurrency-test caveat documented in the test file: convex-test 0.0.43's TransactionManager (node_modules/convex-test/dist/index.js:1268) takes a single `_waitOnCurrentFunction` lock at top-level mutation begin, so `Promise.all` of mutations runs strictly sequentially in the mock. There is NO real OCC retry simulator. Tests therefore prove the FINAL-STATE INVARIANT — even when the second mutation runs back-to-back against the post-winner state, cap/idempotency/meta-parity hold. In production the Convex platform's OCC layer turns the same final-state invariant into the cap-bypass guarantee. See memory `convex-occ-retry-vs-app-cas-conflict- different-layers` for the layer separation. Test helper `seedFollowedCountries(t, userId, codes)` added to seed both the rows AND the user-meta row in parity for tests that bypass the mutation API. Files: - convex/schema.ts: +`followedCountriesUserMeta` table + index. - convex/followedCountries.ts: +`readUserMeta` / `writeUserMeta` helpers, meta read/write inserted into all 3 mutation paths, expanded inline docs on the OCC mechanism. - convex/__tests__/followed-countries-mutations.test.ts: +9 concurrent / parity tests + `seedFollowedCountries` helper + `readUserMetaCount` helper + caveat block on convex-test's serialized mock. Verification: `npm run typecheck` ✅, `npm run typecheck:api` ✅, `npm run test:convex` 292 / 292 ✅. * fix(followed-countries): pre-seeded sharded lock — close nested TOCTOU on user-meta create (P0 v2) Codex round-4 review of PR #3621 caught that the round-3 P0 fix (followedCountriesUserMeta per-user lock) did not actually close the cap-bypass: the meta document is created LAZILY on first mutation, and Convex per-document OCC tracks reads at the document level, not at the empty-index-range level. Two parallel first-ever mutations from the same brand-new user could both read meta=undefined and both INSERT, producing duplicate meta rows that break the next .unique() read AND re-open cap-bypass / counter double-increment. Fix: Approach B (pre-seeded sharded lock). - New table followedCountriesShards with one row per shard id in [0, SHARD_COUNT) (SHARD_COUNT=64 in convex/constants.ts), pre-seeded by _seedShards. - convex/lib/shards.ts::userIdToShard(userId) — deterministic djb2 hash. Frozen contract; changing it would silently remap users. - Every followCountry / unfollowCountry / mergeAnonymousLocal mutation reads its shard at the top (throws SHARDS_NOT_SEEDED loud if missing — operator error, never silent), then patches lastTouchedAt at the end of any non-idempotent path. The read+write pair on an ALREADY-EXISTING document is what triggers Convex OCC to serialize concurrent same-user mutations. - Tier 2 (existing user-meta row) kept additionally for the O(1) cap-check denominator and parity invariant — but its lazy create is now race-free under the shard lock. - Daily cron followed-countries-shards-seed at 03:00 UTC re-runs _seedShards (idempotent) so a missed deploy-seed step self-heals. - Public seedShards mutation for operator CLI: npx convex run --prod followedCountries:seedShards after a fresh deploy without waiting for the cron. Tests added (mutations test file, +6 tests, 292 -> 298): - first-ever follow on a brand-new user creates exactly 1 meta row - two back-to-back mergeAnonymousLocal calls on a brand-new user -> one meta row, no duplicates - operator running _seedShards after partial seed completes idempotently (0 steady-state, plugs holes only) - SHARDS_NOT_SEEDED throws when shards table is empty (operator error path, all three mutations) - public seedShards mutation reachable via operator CLI surface - userIdToShard determinism + range invariant Test fixtures (makeT() helper) call _seedShards before any mutation runs, mirroring the production deploy + cron post-condition. Files changed: - convex/constants.ts: SHARD_COUNT=64 - convex/schema.ts: followedCountriesShards table + index - convex/lib/shards.ts (new): userIdToShard djb2 - convex/followedCountries.ts: shard read/write at every mutation, _seedShards (internal) + seedShards (public operator) mutations - convex/crons.ts: daily _seedShards cron at 03:00 UTC - convex/__tests__/followed-countries-mutations.test.ts: makeT() helper, 6 new tests - convex/__tests__/followed-countries-queries.test.ts: makeT() helper (queries-test invokes followCountry, also needs shards) References: Codex review run /private/tmp/worldmonitor-pr3621-review/ findings_round4.md (P0 v2). Memory: convex-occ-retry-vs-app-cas-conflict-different-layers (layer separation: this is the app-side serialization layer that lets Convex OCC do its job). * fix(followed-countries): treat UNAUTHENTICATED as transient in handoff retry (P1) Codex round-4 P1: subscribeAuthState emits the current signed-in state IMMEDIATELY on subscribe, but Convex auth is not yet ready (the JWT has not been attached to the Convex client at that tick). mergeAnonymousLocal fires before Convex sees the auth -> throws ConvexError({kind:'UNAUTHENTICATED'}). The previous classification (Phase-2 P1 #3) put UNAUTHENTICATED in the PERMANENT-error list alongside INPUT_TOO_LARGE / EMPTY_INPUT, so every transient auth lag cleared localStorage and lost the anonymous follows. Two-part fix: (a) Treat UNAUTHENTICATED as TRANSIENT, not permanent. _runHandoff catch path no longer routes UNAUTHENTICATED to failed-permanent + removeLocalStorage. It falls through to _markFailedAndScheduleRetry, which arms the visibilitychange retry and counts toward MAX_HANDOFF_RETRIES (5). A genuinely-stuck auth mismatch eventually flips to failed-permanent after the budget is exhausted, same as the network-failure path. (b) Defer the merge until Convex auth is ready. waitForConvexAuth() exists at src/services/convex-client.ts:79 — it resolves when Convex's setAuth callback confirms the client is authenticated, with a 10s timeout. We import it and await it BEFORE the mergeAnonymousLocal call so the typical race never fires at all. On timeout we still attempt the call; the catch from (a) treats any resulting UNAUTHENTICATED as transient and the visibilitychange retry wins once Convex catches up. Test seam: _waitForConvexAuthFn module-level binding + new _setDepsForTests({waitForConvexAuth}) override so tests can drive the deferred-by-auth flow without going through the real Convex client. Tests added (tests/followed-countries-sign-in-handoff.test.mjs, +3 new tests, 1 modified — 37 -> 40 in this file): - first call throws UNAUTHENTICATED, visibility retry succeeds -> final state has merged data, localStorage cleared, no follows lost (the canonical scenario this fix targets) - UNAUTHENTICATED IS counted toward MAX_HANDOFF_RETRIES — 5 consecutive UNAUTHENTICATED throws -> failed-permanent (proves runaway-retry guard intact for genuinely-stuck auth) - waitForConvexAuth is awaited BEFORE the merge call (proves deferred-by-auth path is wired correctly) Modified: the original "UNAUTHENTICATED -> 'failed-permanent'; localStorage cleared" test is rewritten to assert the new behavior (state='failed', retry armed, localStorage retained) with an inline comment explaining the previous behavior was wrong. Files changed: - src/services/followed-countries.ts: import waitForConvexAuth from convex-client, _waitForConvexAuthFn seam, await before merge, drop UNAUTHENTICATED from permanent-error branch - tests/followed-countries-sign-in-handoff.test.mjs: 1 modified + 3 new tests References: Codex review run /private/tmp/worldmonitor-pr3621-review/ findings_round4.md (P1). waitForConvexAuth helper found at src/services/convex-client.ts:79 — exists today and is used by the existing entitlement subscription path; this fix wires it into the followed-countries handoff for the same reason. * fix(ci): seed followedCountries shards on Convex deploy (P1) The followCountry / unfollowCountry / mergeAnonymousLocal mutations throw SHARDS_NOT_SEEDED if followedCountriesShards is empty. Today the seed runs only via the 03:00 UTC daily cron, so a deploy landing at 04:00 UTC would leave the feature broken for ~23h until the next cron tick. Run npx convex run --prod followedCountries:_seedShards inline after npx convex deploy --yes so the table is populated before any traffic hits the new mutations. Idempotent: existing shard rows are skipped, only missing ids in [0, SHARD_COUNT) are inserted. * fix(followed-countries): race-tolerant shard seed + dedupe cron + remove public seed (P1) Two stacked P1 issues from Codex round-3 review of PR #3621: 1. Public seedShards mutation was unauthenticated — any browser ConvexHttpClient could call it. Removed; the post-deploy CI step now targets the internal _seedShards directly via `npx convex run --prod followedCountries:_seedShards` (npx convex run resolves internal functions by file:export path). 2. _seedShards has a TOCTOU race: two simultaneous calls against an empty table both read empty, both insert the full range, producing 2 rows per shardId. Previously readShardOrThrow used .unique() which throws on duplicates → bricks the affected shard for all users hashing to it. Approach D (real-world correct): make readShardOrThrow tolerant of duplicates via .first() (returns oldest by _creationTime tiebreaker, so OCC contention is preserved across all in-flight mutations on that shard during the duplicate window) AND add a daily _dedupeShards cron that deletes extras keeping the oldest row per shardId. Tests: - duplicate shard rows: mutation succeeds, counter parity holds - _dedupeShards: zero dups → no-op; N dups → reduces to 1 row per shardId, oldest survives - _seedShards idempotent under back-to-back concurrent re-run - public seedShards no longer exported (source-text negative assertion) Net: 298 → 302 tests, all green. * feat(follow-button): minimal CSS for star button + host layouts (PR A foundation) Third-pass review P2: PR A's src/utils/follow-button.ts:220 emits .wm-follow-btn* markup mounted on three live surfaces (CIIPanel, CountryDeepDivePanel, CountryIntelModal) but no CSS shipped — buttons rendered as native browser controls and the CIIPanel host (a span inside a block-layout .cii-country row) put the star on its own line. Visual analogue: .cii-share-btn (transparent + 1px var(--border) + var(--text-muted) text + var(--semantic-info) hover). Same border-radius family, same micro-padding scale. CSS variables used: --border, --border-strong, --text-muted, --semantic-info. Reuses the existing @Keyframes spin. Critical layout fix (CIIPanel): added position:relative to .cii-country and absolute-positioned .cii-follow-btn-host at top:6px left:6px. The :not(:empty) gate keeps the rule a no-op when the feature flag is off (handle.html === ''), and the sibling-combinator rule .cii-follow-btn-host:not(:empty) ~ .cii-header { padding-left:26px } shifts header content right ONLY when the star is mounted — flag-off rows render identically to today. CDP + CountryIntelModal hosts are simple display:inline-flex since both parent containers (.cdp-header-left, .country-intel-title) are already flex with gap. Polish (color tuning, hover transitions, animation curves, empty/cap nudge styling) intentionally deferred to PR B per the third-pass review. +156 lines (single appended block in src/styles/main.css). * fix(followed-countries): serialize country counters * fix(followed-countries): register picker global
This was referenced Jul 1, 2026
koala73
pushed a commit
that referenced
this pull request
Jul 6, 2026
…vider timeout + harden classification (#4980 review) Addresses the ce-code-review findings on #4980: - #1/#4: raise MARKET_IMPLICATIONS_MIN_RUN_BUDGET_MS 20_000 -> 30_000 (= max provider.timeout 25s + FORECAST_LLM_STAGE_BUDGET_GUARD_MS 5s). At 20s the pre-call guard admitted calls that were then timeout-CAPPED below the provider's own timeout, which is indistinguishable from a genuinely hung provider: a real openrouter timeout in the [20s,30s) band was misclassified as a benign budget starve and its SEED_ERROR suppressed, and 20-25s calls were guaranteed to abort mid-flight (~15s wasted). Admitting only at >=30s gives the provider its full window, so any timeout is attributable and a genuine failure still surfaces SEED_ERROR. - #2: getRemainingForecastLlmBudgetMs now delegates the run-remaining calc to getRemainingForecastLlmRunBudgetMs (was a verbatim duplicate formula). - #3: document the failure-reason classification invariant on the sawProviderFailure / sawBudgetCappedTimeout declarations and the budget-capped-timeout heuristic (comments only, no behavior change). - #5: add tests for the 20-30s guard skip (locks the 30s threshold), the retained mid-call budget_exhausted preserve branch, and its provider_failed symmetry (via the __setForecastLlmCallOverrideForTests seam). - #6: test 1 now sets a real provider key + transport counter so its llmCalls===0 tripwire exercises the guard instead of the !apiKey short-circuit; removed the now-unused budgetStarveFetch helper. test 4 deadline bumped 25s -> 35s to keep passing the new 30s guard. Tests: 605 pass across all seed-forecasts.mjs consumers; biome + unicode clean. Claude-Session: https://claude.ai/code/session_017odw3Pf9ue8RZzxYAQ37P8
koala73
added a commit
that referenced
this pull request
Jul 6, 2026
…ved of the shared LLM run budget (#4978) (#4980) * fix(forecast): stop false SEED_ERROR when market_implications is starved of the shared LLM run budget (#4978) market_implications is the LAST forecast LLM stage (afterPublish) and shares the single 150s run budget with every upstream stage. When upstream stages are slow (e.g. deepseek-v4-flash breaching its 25s call timeout on combined/ scenario, #4944), they drain that budget before this tail stage runs; callForecastLLM then throws a budget error and returns null. Pre-fix the caller treated that starve identically to a real LLM failure and wrote a status:'error' seed-meta, so /api/health flipped to SEED_ERROR for benign, self-healing resource contention (observed 2026-07-06 14:03; self-healed 14:15). Two changes: - Distinguish a budget-starve from a real failure. On starve (pre-call guard, or a null result with the run budget now <=0), preserve last-good WITHOUT rewriting seed-meta.fetchedAt, so age-based STALE_SEED (maxStaleMin=120) still escalates if the starve persists past 2h (gpsjam preserve-last-good design). Genuine provider failures with budget remaining still surface SEED_ERROR. - Reorder: run market_implications BEFORE the best-effort telemetry in afterPublish (history + deep-forecast snapshots, ~20s R2 trace export) so that wall-clock can't push the tail stage past the run deadline. This recovers the ~20s that pushed it over the 150s deadline in the failing run. Root trigger (deepseek-v4-flash latency draining the shared budget) is #4944 territory; a complementary combined-stage groq fallback is a 1-line Railway env change (FORECAST_LLM_COMBINED_PROVIDER_ORDER=openrouter,groq), left out of code to respect the intentional #4944 pin. Tests: tests/market-implications-budget-starve.test.mjs (starve preserves last-good + no error meta; genuine failure still errors) + an afterPublish ordering guard in market-implications-seed-health.test.mjs. Failing-test-first per Bug Fix Protocol. Claude-Session: https://claude.ai/code/session_019uhnqAKEHTws3QMaUvq58N * fix(forecast): preserve market implication failure signals Address PR #4980 review feedback by keeping provider failures distinct from run-budget starvation and restoring stale OK market-implications meta from last-good payloads during starved runs. Also updates the budget-starve tests so the provider-failure regression performs a real provider request. * fix(forecast): raise market_implications budget guard to the full provider timeout + harden classification (#4980 review) Addresses the ce-code-review findings on #4980: - #1/#4: raise MARKET_IMPLICATIONS_MIN_RUN_BUDGET_MS 20_000 -> 30_000 (= max provider.timeout 25s + FORECAST_LLM_STAGE_BUDGET_GUARD_MS 5s). At 20s the pre-call guard admitted calls that were then timeout-CAPPED below the provider's own timeout, which is indistinguishable from a genuinely hung provider: a real openrouter timeout in the [20s,30s) band was misclassified as a benign budget starve and its SEED_ERROR suppressed, and 20-25s calls were guaranteed to abort mid-flight (~15s wasted). Admitting only at >=30s gives the provider its full window, so any timeout is attributable and a genuine failure still surfaces SEED_ERROR. - #2: getRemainingForecastLlmBudgetMs now delegates the run-remaining calc to getRemainingForecastLlmRunBudgetMs (was a verbatim duplicate formula). - #3: document the failure-reason classification invariant on the sawProviderFailure / sawBudgetCappedTimeout declarations and the budget-capped-timeout heuristic (comments only, no behavior change). - #5: add tests for the 20-30s guard skip (locks the 30s threshold), the retained mid-call budget_exhausted preserve branch, and its provider_failed symmetry (via the __setForecastLlmCallOverrideForTests seam). - #6: test 1 now sets a real provider key + transport counter so its llmCalls===0 tripwire exercises the guard instead of the !apiKey short-circuit; removed the now-unused budgetStarveFetch helper. test 4 deadline bumped 25s -> 35s to keep passing the new 30s guard. Tests: 605 pass across all seed-forecasts.mjs consumers; biome + unicode clean. Claude-Session: https://claude.ai/code/session_017odw3Pf9ue8RZzxYAQ37P8 --------- Co-authored-by: Elie Habib <[email protected]>
koala73
added a commit
that referenced
this pull request
Jul 26, 2026
… mutation-proof Code review of #5605 found the outcome signal wrong in both directions and all three new regression guards passing with the bug restored (proven by execution). Addresses every finding. Classification (#1, #5, #8) — `lastAttemptedProvider === 'none'` conflated causes: - too quiet: canAttemptServerSummarization() denies both an anon/free principal AND an entitled one inside the 403/429 cooldown, so a paying user's live entitlement outage was demoted to console.debug for up to 15 min (24 h via Retry-After). trackLLMFailure is a no-op and no captureConsoleIntegration exists, so that was the only signal — the #5600 shape. summarize-gate now exports isServerSummarizationSuppressed() and the chain records which denial it hit. - too loud: CircuitBreaker.execute returns its default WITHOUT running the callback while on cooldown (breaker defaults maxFailures=2, cooldown=5min), so a chain that contacted nobody still reported "All providers failed". Marking now happens INSIDE the breaker callback, and a short-circuit is recorded as its own cause. - the quiet path no longer claims "using designed fallback"; reaching it means browser T5 never ran and callers render an error/unavailable state. Guards (#2, #3, #6) — each of these was green with the bug restored: - indexOf matched the gate name inside a COMMENT, so moving the mark above the gate passed. Source assertions now strip comments; more importantly the mark lives inside the breaker callback, making the ordering structural. - the locality assertion ran against the whole file, so hoisting the state to module scope passed. Now scoped to generateSummary with a creation-count pin and a broadened module-global check. - the raw-warn regex only matched quoted literals while this file's idiom is a template literal. Now delimiter-agnostic. - slice anchors are asserted to resolve; a renamed anchor made slice(a, -1) silently widen to end-of-file. The decision surface moved into a pure classifier covered by an executed truth table rather than grep. Verified: all three original bypasses now fail (32/32 green, 31/32 with each mutation applied); typecheck and biome clean. Claude-Session: https://claude.ai/code/session_018XDm48Kzuv1GQe4PR1qimE
koala73
added a commit
that referenced
this pull request
Jul 26, 2026
… mutation-proof Code review of #5605 found the outcome signal wrong in both directions and all three new regression guards passing with the bug restored (proven by execution). Addresses every finding. Classification (#1, #5, #8) — `lastAttemptedProvider === 'none'` conflated causes: - too quiet: canAttemptServerSummarization() denies both an anon/free principal AND an entitled one inside the 403/429 cooldown, so a paying user's live entitlement outage was demoted to console.debug for up to 15 min (24 h via Retry-After). trackLLMFailure is a no-op and no captureConsoleIntegration exists, so that was the only signal — the #5600 shape. summarize-gate now exports isServerSummarizationSuppressed() and the chain records which denial it hit. - too loud: CircuitBreaker.execute returns its default WITHOUT running the callback while on cooldown (breaker defaults maxFailures=2, cooldown=5min), so a chain that contacted nobody still reported "All providers failed". Marking now happens INSIDE the breaker callback, and a short-circuit is recorded as its own cause. - the quiet path no longer claims "using designed fallback"; reaching it means browser T5 never ran and callers render an error/unavailable state. Guards (#2, #3, #6) — each of these was green with the bug restored: - indexOf matched the gate name inside a COMMENT, so moving the mark above the gate passed. Source assertions now strip comments; more importantly the mark lives inside the breaker callback, making the ordering structural. - the locality assertion ran against the whole file, so hoisting the state to module scope passed. Now scoped to generateSummary with a creation-count pin and a broadened module-global check. - the raw-warn regex only matched quoted literals while this file's idiom is a template literal. Now delimiter-agnostic. - slice anchors are asserted to resolve; a renamed anchor made slice(a, -1) silently widen to end-of-file. The decision surface moved into a pure classifier covered by an executed truth table rather than grep. Verified: all three original bypasses now fail (32/32 green, 31/32 with each mutation applied); typecheck and biome clean. Claude-Session: https://claude.ai/code/session_018XDm48Kzuv1GQe4PR1qimE
koala73
added a commit
that referenced
this pull request
Jul 26, 2026
…by-design chains (#5377) (#5605) * fix(summarization): stop logging 'All providers failed' for declined-by-design chains (#5377) Both chain-exhausted sites (BETA + normal mode) warned 'All providers failed' even when nothing was attempted — the entitlement gate (#4913/#4915) had declined every server dispatch and browser T5 was unavailable, which is the designed anonymous path. The outage-shaped warning is indistinguishable from a real provider failure in console triage and cost the #5377 investigation two wrong hypotheses. Route both sites through logChainOutcome(): when lastAttemptedProvider is still 'none' (tryApiProvider marks attempts only after the feature + entitlement gates; tryBrowserT5 only after the mlWorker availability check) log at debug with an accurate 'skipped: no eligible provider' message; reserve warn for chains where a provider was genuinely attempted and failed. Note: #5377's part 1 (the enqueue gate — zero anon summarize dispatches) already landed in #4915 via configureSummarizeGate(hasPremiumAccess) and is pinned by tests/summarize-entitlement-gate; this closes the remaining acceptance item. Tests: extended tests/summarize-entitlement-gate.test.mts — the fail sites must route through logChainOutcome (no unconditional string-literal 'All providers failed' warn), debug-before-warn branch on 'none', and attempt marking must stay behind the gates so 'none' means declined-by-design. 21/21 green; RED (1 fail) against the pre-fix source. tsc clean. Co-Authored-By: Claude Fable 5 <[email protected]> * fix(summarization): isolate provider attempt tracking * docs: update service module count * fix(review): make the #5377 outcome classifier correct and its guards mutation-proof Code review of #5605 found the outcome signal wrong in both directions and all three new regression guards passing with the bug restored (proven by execution). Addresses every finding. Classification (#1, #5, #8) — `lastAttemptedProvider === 'none'` conflated causes: - too quiet: canAttemptServerSummarization() denies both an anon/free principal AND an entitled one inside the 403/429 cooldown, so a paying user's live entitlement outage was demoted to console.debug for up to 15 min (24 h via Retry-After). trackLLMFailure is a no-op and no captureConsoleIntegration exists, so that was the only signal — the #5600 shape. summarize-gate now exports isServerSummarizationSuppressed() and the chain records which denial it hit. - too loud: CircuitBreaker.execute returns its default WITHOUT running the callback while on cooldown (breaker defaults maxFailures=2, cooldown=5min), so a chain that contacted nobody still reported "All providers failed". Marking now happens INSIDE the breaker callback, and a short-circuit is recorded as its own cause. - the quiet path no longer claims "using designed fallback"; reaching it means browser T5 never ran and callers render an error/unavailable state. Guards (#2, #3, #6) — each of these was green with the bug restored: - indexOf matched the gate name inside a COMMENT, so moving the mark above the gate passed. Source assertions now strip comments; more importantly the mark lives inside the breaker callback, making the ordering structural. - the locality assertion ran against the whole file, so hoisting the state to module scope passed. Now scoped to generateSummary with a creation-count pin and a broadened module-global check. - the raw-warn regex only matched quoted literals while this file's idiom is a template literal. Now delimiter-agnostic. - slice anchors are asserted to resolve; a renamed anchor made slice(a, -1) silently widen to end-of-file. The decision surface moved into a pure classifier covered by an executed truth table rather than grep. Verified: all three original bypasses now fail (32/32 green, 31/32 with each mutation applied); typecheck and biome clean. Claude-Session: https://claude.ai/code/session_018XDm48Kzuv1GQe4PR1qimE * chore(docs): regenerate stats after rebase onto main Claude-Session: https://claude.ai/code/session_018XDm48Kzuv1GQe4PR1qimE --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Elie Habib <[email protected]>
benngee85
pushed a commit
to benngee85/VISTA
that referenced
this pull request
Jul 26, 2026
…by-design chains (koala73#5377) (koala73#5605) * fix(summarization): stop logging 'All providers failed' for declined-by-design chains (koala73#5377) Both chain-exhausted sites (BETA + normal mode) warned 'All providers failed' even when nothing was attempted — the entitlement gate (koala73#4913/koala73#4915) had declined every server dispatch and browser T5 was unavailable, which is the designed anonymous path. The outage-shaped warning is indistinguishable from a real provider failure in console triage and cost the koala73#5377 investigation two wrong hypotheses. Route both sites through logChainOutcome(): when lastAttemptedProvider is still 'none' (tryApiProvider marks attempts only after the feature + entitlement gates; tryBrowserT5 only after the mlWorker availability check) log at debug with an accurate 'skipped: no eligible provider' message; reserve warn for chains where a provider was genuinely attempted and failed. Note: koala73#5377's part 1 (the enqueue gate — zero anon summarize dispatches) already landed in koala73#4915 via configureSummarizeGate(hasPremiumAccess) and is pinned by tests/summarize-entitlement-gate; this closes the remaining acceptance item. Tests: extended tests/summarize-entitlement-gate.test.mts — the fail sites must route through logChainOutcome (no unconditional string-literal 'All providers failed' warn), debug-before-warn branch on 'none', and attempt marking must stay behind the gates so 'none' means declined-by-design. 21/21 green; RED (1 fail) against the pre-fix source. tsc clean. Co-Authored-By: Claude Fable 5 <[email protected]> * fix(summarization): isolate provider attempt tracking * docs: update service module count * fix(review): make the koala73#5377 outcome classifier correct and its guards mutation-proof Code review of koala73#5605 found the outcome signal wrong in both directions and all three new regression guards passing with the bug restored (proven by execution). Addresses every finding. Classification (koala73#1, koala73#5, koala73#8) — `lastAttemptedProvider === 'none'` conflated causes: - too quiet: canAttemptServerSummarization() denies both an anon/free principal AND an entitled one inside the 403/429 cooldown, so a paying user's live entitlement outage was demoted to console.debug for up to 15 min (24 h via Retry-After). trackLLMFailure is a no-op and no captureConsoleIntegration exists, so that was the only signal — the koala73#5600 shape. summarize-gate now exports isServerSummarizationSuppressed() and the chain records which denial it hit. - too loud: CircuitBreaker.execute returns its default WITHOUT running the callback while on cooldown (breaker defaults maxFailures=2, cooldown=5min), so a chain that contacted nobody still reported "All providers failed". Marking now happens INSIDE the breaker callback, and a short-circuit is recorded as its own cause. - the quiet path no longer claims "using designed fallback"; reaching it means browser T5 never ran and callers render an error/unavailable state. Guards (koala73#2, koala73#3, koala73#6) — each of these was green with the bug restored: - indexOf matched the gate name inside a COMMENT, so moving the mark above the gate passed. Source assertions now strip comments; more importantly the mark lives inside the breaker callback, making the ordering structural. - the locality assertion ran against the whole file, so hoisting the state to module scope passed. Now scoped to generateSummary with a creation-count pin and a broadened module-global check. - the raw-warn regex only matched quoted literals while this file's idiom is a template literal. Now delimiter-agnostic. - slice anchors are asserted to resolve; a renamed anchor made slice(a, -1) silently widen to end-of-file. The decision surface moved into a pure classifier covered by an executed truth table rather than grep. Verified: all three original bypasses now fail (32/32 green, 31/32 with each mutation applied); typecheck and biome clean. Claude-Session: https://claude.ai/code/session_018XDm48Kzuv1GQe4PR1qimE * chore(docs): regenerate stats after rebase onto main Claude-Session: https://claude.ai/code/session_018XDm48Kzuv1GQe4PR1qimE --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Elie Habib <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Description
flow_dropandflow_price_divergencetoSignalTypeand the correlation pipeline.detectPipelineFlowDrops(events)plusincludesKeywordhelper and keyword lists (PIPELINE_KEYWORDS,FLOW_DROP_KEYWORDS) and thresholds (FLOW_PRICE_THRESHOLD,ENERGY_COMMODITY_SYMBOLS) insrc/services/correlation.ts.analyzeCorrelationsand emitflow_dropsignals andflow_price_divergencewhen energy commodities move without pipeline flow news.src/components/SignalModal.tsandsrc/styles/main.cssso they display consistently in the existing signal modal.Testing
npm run dev(Vite) which launched but produced some network proxy errors (ENETUNREACH) and anxdg-openspawn warning during the run.flow_dropandflow_price_divergenceitems into the modal and capture a screenshot, which completed successfully and produced an artifact.Codex Task