fix(feeds): remove dead feed domains from RSS allowed-domains list#1626
Merged
Conversation
PR #1596 removed the feeds but left the domains in the allowlist. The relay still accepted proxy requests for these 403-blocked domains from clients with cached old bundles. Removed: - breakingdefense.com (403) - www.arabnews.com (403) - www.aei.org (403) - mymodernmet.com (403) Updated all 3 copies: shared/, scripts/shared/, api/
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
stevenjmiklovic
added a commit
to stevenjmiklovic/worldmonitor
that referenced
this pull request
Mar 20, 2026
…13) * fix(seeds): increase forecast TTL from 3600s to 4200s (#1625) TTL (1h) equaled cron interval (1h), leaving zero buffer for container cold starts. Health saw EMPTY records=0 during the gap between key expiry and next seed run. 70min TTL covers the cold start window. * fix(feeds): remove dead feed domains from RSS allowed-domains list (#1626) PR #1596 removed the feeds but left the domains in the allowlist. The relay still accepted proxy requests for these 403-blocked domains from clients with cached old bundles. Removed: - breakingdefense.com (403) - www.arabnews.com (403) - www.aei.org (403) - mymodernmet.com (403) Updated all 3 copies: shared/, scripts/shared/, api/ * refactor(seeds): extract USNI fleet seed from relay with proxy support (#1627) * refactor(seeds): extract USNI fleet seed from relay with proxy support USNI's Cloudflare blocks Railway IPs (JA3 fingerprinting), causing the relay's USNI seed to fail with HTML response instead of JSON. Changes: - New seed-usni-fleet.mjs: standalone script with HTTP CONNECT proxy support via RESIDENTIAL_PROXY_AUTH (falls back to direct fetch) - Removed ~240 lines of USNI code from ais-relay.cjs - TTL bumped from 21600s (6h) to 25200s (7h) for cold start buffer - Exact same parsing logic (hull types, region coords, vessel extraction, strike group detection, battle force summary) - runSeed pattern with lock, validation, seed-meta, verification Deploy: Railway cron service, 6h interval, needs RESIDENTIAL_PROXY_AUTH * fix(seeds): only use RESIDENTIAL_PROXY_AUTH for USNI proxy * fix(seeds): use OREF_PROXY_AUTH for USNI proxy (the one proxy available) * fix(supply-chain): remount transit chart on no-op renders (#1630) * refactor: share edge Turnstile helper (#1628) * refactor: dedupe signals upstream fetch headers and timeout (#1629) * fix(seeds): use curl for USNI fetch (JA3 fingerprint bypass) (#1631) Node.js TLS fingerprint gets blocked by Cloudflare. curl's fingerprint passes. Strategy: curl direct → curl+proxy → Node.js direct. Same pattern as relay's orefCurlFetch. * perf(ci): skip Vercel deploy when only scripts/ change on main (#1633) Previously, every merge to main triggered a Vercel build even for scripts-only changes (seed scripts, relay updates). Now checks if any web-relevant files changed on main too, skipping the build when only scripts/, docs/, .github/, etc. are modified. * test(seeds): cover warm-ping origin header (#1632) * fix(usni-fleet): add Node.js proxy fallback for Cloudflare bypass (#1635) * fix(usni-fleet): add Node.js HTTP CONNECT proxy fallback, detect Cloudflare HTML curl is not available in Railway's Railpack v0.18.0 containers. The seed was failing with ENOENT on curl, then getting Cloudflare-blocked on Node.js direct. - Add fetchViaHttpProxy: Node.js HTTP CONNECT tunnel through residential proxy (no curl dependency). Uses the same RESIDENTIAL_PROXY_AUTH env. - Add Cloudflare HTML detection: reject early when response starts with <!DOCTYPE instead of passing HTML to JSON.parse. - Fallback chain: curl direct -> curl+proxy -> Node.js+proxy -> Node.js direct - Add nixpacks.toml with curl for future Railpack builds * fix: use ESM import for node:http (require breaks in .mjs) * test: rewrite transit chart test as structural contract verification (#1636) Replace fragile source-string extraction + new Function() compilation with structural pattern checks on the source code. Tests verify: - render() clears chart before content change - clearTransitChart() cancels timer, disconnects observer, destroys chart - MutationObserver setup for DOM readiness detection - Fallback timer for no-op renders (100-500ms range) - Both callbacks (observer + timer) clean up each other - Tab switch and collapse clear chart state - Mount function guards against missing element/data Replaces PR #1634's approach which was brittle (method body extraction, TypeScript cast stripping, sandboxed execution). * fix: log fetch error cause in seed retry/FATAL handlers (#1638) * test: rewrite transit chart test as structural contract verification Replace fragile source-string extraction + new Function() compilation with structural pattern checks on the source code. Tests verify: - render() clears chart before content change - clearTransitChart() cancels timer, disconnects observer, destroys chart - MutationObserver setup for DOM readiness detection - Fallback timer for no-op renders (100-500ms range) - Both callbacks (observer + timer) clean up each other - Tab switch and collapse clear chart state - Mount function guards against missing element/data Replaces PR #1634's approach which was brittle (method body extraction, TypeScript cast stripping, sandboxed execution). * fix: log fetch error cause in seed retry and FATAL handlers Node 20 fetch() throws TypeError('fetch failed') with the real error hidden in err.cause (DNS, TLS, timeout). The current logging only shows 'fetch failed' which is useless for diagnosis. Now logs err.cause.message in both withRetry() retries and FATAL catch blocks. * fix(railway): keep Node 20 + force IPv4 DNS ordering (#1641) Node 20's fetch() (undici) tries IPv6 first. Railway containers don't support IPv6 (IPV6_NDISC failures in network trace), causing all seed services to crash. Fix: set NODE_OPTIONS=--dns-result-order=ipv4first via nixpacks.toml so all Railway services prefer IPv4. Keeps Node 20 for import attributes. * refactor: dedupe contact api json responses (#1639) * refactor(llm): consolidate provider chain to single source of truth (#1640) * fix(relay): add LLM fallback chain to ais-relay classify Replace single Groq-only LLM call with provider fallback chain (Groq → OpenRouter → Ollama) matching seed-insights.mjs pattern. If Groq fails or is unavailable, classify falls through to the next configured provider automatically. * refactor(llm): consolidate provider chain to single source of truth - Fix OpenRouter model: openrouter/free → google/gemini-2.5-flash in canonical llm.ts - Migrate 4 intelligence handlers (classify-event, batch-classify, deduct-situation, get-country-intel-brief) from hardcoded Groq-only to callLlm() with full ollama → groq → openrouter fallback chain - Remove duplicate getProviderCredentials from news/v1/_shared.ts, re-export canonical - Remove orphaned GROQ_API_URL/GROQ_MODEL from intelligence/v1/_shared.ts - Reorder script provider chains (ais-relay.cjs, seed-insights.mjs) to canonical ollama → groq → openrouter order - Net -161 lines: eliminated duplicated provider logic across 9 files * fix: eliminate double JSON parse in classify-event, throw on runSeed verification failure * fix(tests): add llm module alias to country-intel-brief test fixture * fix: preserve generic LLM_API_* fallback, add retry to seed verification - Add 'generic' provider to callLlm() chain for LLM_API_URL/LLM_API_KEY/LLM_MODEL (preserves existing OpenAI-compatible endpoint contract) - Change seed verification to warn-only with 1 retry instead of fatal throw (write already succeeded, transient read failure shouldn't fail the job) - Update docs to reflect new provider fallback chain * feat(advisories): gold standard migration for security advisories (#1637) * feat(advisories): gold standard migration for security advisories Move security advisories from client-side RSS fetching (24 feeds per page load) to Railway cron seed with Redis-read-only Vercel handler. - Add seed script fetching via relay RSS proxy with domain allowlist - Add ListSecurityAdvisories proto, handler, and RPC cache tier - Add bootstrap hydration key for instant page load - Rewrite client service: bootstrap -> RPC fallback, no browser RSS - Wire health.js, seed-health.js, and dataSize tracking * fix(advisories): empty RPC returns ok:true, use full country map P1 fixes from Codex review: - Return ok:true for empty-but-successful RPC responses so the panel clears to empty instead of stuck loading on cold environments - Replace 50-entry hardcoded country map with 251-entry shared config generated from the project GeoJSON + aliases, matching coverage of the old client-side nameToCountryCode matcher * fix(advisories): add Cote d'Ivoire and other missing country aliases Adds 14 missing aliases including "cote d ivoire" (US State Dept title format), common article-prefixed names (the Bahamas, the Gambia), and alternative official names (Czechia, Eswatini, Cabo Verde, Timor-Leste). * fix(proto): inject @ts-nocheck via Makefile generate target buf generate does not emit @ts-nocheck, but tsc strict mode rejects the generated code. Adding a post-generation sed step in the Makefile ensures both CI proto-freshness (make generate + diff) and CI typecheck (tsc --noEmit) pass consistently. * fix(predictions): remove Kalshi API key, enforce Redis-only on Vercel (#1642) Kalshi public market data endpoints require no authentication. Remove the KALSHI_API_KEY gate that was disabling Kalshi entirely when the env var was missing, and drop the Authorization header. Rewrite the Vercel RPC handler to read from Railway-seeded Redis only (gold standard), removing the fallback that fetched directly from Gamma/Kalshi APIs on Vercel edge. Handler goes from 330 to 85 lines. Double all prediction timing values to reduce Railway cron cost: - Redis TTL: 15min -> 30min - Health maxStaleMin: 15min -> 30min - Client hydration freshness: 20min -> 40min - Railway cron: 10min -> 20min (requires dashboard update) * fix: add fetch error cause logging to all remaining seed scripts (#1643) * fix(seeds): improve resilience and fix dead APIs across seed scripts (#1644) * fix(seeds): improve resilience and fix dead APIs across seed scripts - Fix wrong domain in seed-service-statuses (worldmonitor.app to api.worldmonitor.app) - Fix Kalshi API domain migration (trading-api.kalshi.com to api.elections.kalshi.com) - Replace dead trending APIs (gitterapp.com, herokuapp.com) with OSSInsight + GitHub Search - Fix case-sensitive HTML detection in seed-usni-fleet (lowercase doctype not matched) - Add Promise.allSettled rejection logging across 8 seed scripts - Wrap fetch loops in try-catch (seed-supply-chain-trade, seed-economy) so a single network error no longer kills the entire function - Update list-trending-repos.ts RPC handler to match seed changes * fix(seeds): correct OSSInsight response parsing and period-aware GitHub Search fallback - OSSInsight returns {data: {rows: [...]}} not {data: [...]}, fix both seed and handler - GitHub Search fallback now respects period parameter (daily=1d, weekly=7d, monthly=30d) * fix(seeds): correct OSSInsight period values (past_week/past_month, not past_7_days/past_28_days) * Add Greek news channels & feed (#1602) * Add Greek news channels * Add ERT and SKAI hlsUrl to LiveNewsPanel.ts --------- Co-authored-by: Elie Habib <[email protected]> * feat(scoring): consume live advisory data from Redis with hardcoded fallback (#1620) Read live travel advisory levels from intelligence:advisories:v1 Redis key (populated by seed-security-advisories.mjs). Fall back to the original hardcoded ADVISORY_LEVELS_FALLBACK map when Redis data is unavailable. Closes #1359. * fix(sentry): tighten noise filters for 6 unresolved issues (#1645) Fix 2 regexes that let errors slip through: - AbortError filter now handles value without type prefix - Qt() filter now matches Qt(...) with arguments Add 4 new filters for browser-specific noise: - ES module syntax errors (Pale Moon, QQ Browser) - ucConfig (UC Browser V8 format) - getShaderPrecisionFormat (WebGL context loss on iOS) * refactor: dedupe register-interest json responses (#1647) * fix(usni): move USNI fleet seed back to AIS relay (fixed IP for Froxy proxy) (#1648) The standalone seed-usni-fleet.mjs cannot reach USNI because: 1. USNI Cloudflare blocks Node.js TLS fingerprint (JA3) 2. curl is not installed on Railway cron containers 3. Froxy residential proxy is IP-whitelisted to the relay fixed IP Move the USNI seed loop back into ais-relay.cjs where it has access to curl + the whitelisted proxy. Uses orefCurlFetch for the fetch, same pattern as the OREF alerts loop. Writes to the same Redis keys (usni-fleet:sebuf:v1, stale:v1, seed-meta:military:usni-fleet). 6h seed interval, 7h TTL, 7d stale TTL (unchanged from standalone). * fix(seeds): allow conflict-intel seed to succeed without ACLED keys (#1651) Validation now accepts empty ACLED events array when humanitarian or pizzint data was fetched. Previously the seed wrote extra keys (humanitarian, pizzint) but skipped the canonical key because validateFn required non-empty events. * fix(data): restore bootstrap and cache test coverage (#1649) * fix(data): restore bootstrap and cache test coverage * fix: resolve linting and test failures - Remove dead writeSeedMeta/estimateRecordCount functions from redis.ts (intentionally removed from cachedFetchJson; seed-meta now written only by explicit seed flows, not generic cache reads) - Fix globe dayNight test to match actual code (forces dayNight: false + hideLayerToggle, not catalog-based exclusion) - Fix country-geometry test mock URL from CDN to /data/countries.geojson (source changed to use local bundled file) * fix(lint): remove duplicate llm-health key in redis-caching test Duplicate object key '../../../_shared/llm-health' caused the stub to be overwritten by the real module. Removed the second entry so the test correctly uses the stub. * feat(forecast): add structured scenario pipeline and trace export (#1646) * feat(forecast): add AI Forecasts prediction module (Pro-tier) MiroFish-inspired prediction engine that generates structured forecasts across 6 domains (conflict, market, supply chain, political, military, infrastructure) using existing WorldMonitor data streams. - Proto definitions for ForecastService with GetForecasts RPC - Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors, cross-domain cascade resolver, prediction market calibration, and trend detection via prior snapshot comparison - Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement) - Lazy-loaded ForecastPanel with domain filters, probability bars, trend arrows, signal evidence, and cascade links - Health monitoring integration (seed-meta freshness tracking) - Refresh scheduler with API key guard * test(forecast): add 47 unit tests for forecast detectors and utilities Covers forecastId, normalize, resolveCascades, calibrateWithMarkets, computeTrends, and smoke tests for all 6 domain detectors. Exports testable functions from seed script with direct-run guard. * fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category - Seed script used 'infra' but ForecastPanel filtered on 'infrastructure', causing Infra tab to show zero results - Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP * fix(forecast): move CSS to one-time injection, improve type safety - P2: Move style block from setContent to one-time document.head injection to prevent CSS accumulation on repeated renders - P3: Replace +toFixed(3) with Math.round for readability in seed script - P3: Use Forecast type instead of any[] in RPC handler filter * fix(forecast): handle sebuf proto data shapes from Redis Detectors now normalize CII scores from server-side proto format (combinedScore, TREND_DIRECTION_RISING, region) to uniform shape. Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH). Added confidence floor of 0.3 for single-source predictions. Verified against live Redis: 2 predictions generated (Iran infra shutdown, IL political instability). * feat(forecast): unlock AI Forecasts on web, lock desktop only (trial) - Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free) - Panel locked on desktop only (same as oref-sirens/telegram-intel) - Remove API key guards from data-loader and refresh scheduler - Web users get full access during trial period * chore: regenerate proto types with make generate Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped @ts-nocheck from output, added it back to all 50 generated files. Fixed 4 type errors from proto codegen changes: - MarketSource enum -> string union type - TemporalAnomalyProto -> TemporalAnomaly rename - webcam lastUpdated number -> string * chore: add proto freshness check to pre-push hook Runs make generate before push and compares checksums of generated files. If proto types are stale, blocks push with instructions to regenerate. Skips gracefully if buf CLI is not installed. * fix(forecast): use chokepoints v4 key, include ciiContribution in unrest - P1: Switch chokepoints input from stale v2 to active v4 Redis key, matching bootstrap.js and cache-keys.ts - P2: Add ciiContribution to unrest component fallback chain in normalizeCiiEntry so political detector reads the correct sebuf field * feat(forecast): Phase 2 LLM scenario enrichment + confidence model MiroFish-inspired enhancements: - LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric adjustment). Evidence-grounded prompts with mandatory signal citation and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE. - Top-4 predictions batched into single LLM call for cost efficiency. - News context from newsInsights attached to all predictions for LLM prompt grounding (NOT in signals, cannot affect confidence). - Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE mapping (deduplicates cii+cii_delta, theater+indicators) + calibration agreement from prediction market drift. Floor 0.2, ceiling 1.0. - Output validation: rejects scenarios without signal references. - Truncated JSON repair for small model output. - Structured JSON logging for LLM calls. - Redis cache for LLM scenarios (1h TTL). - 23 new tests (70 total), all passing. - Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded scenario narratives from real WorldMonitor data. * feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades MiroFish-inspired enhancements: - Multi-perspective LLM analysis: top-2 predictions get strategic, regional, and contrarian viewpoints via combined LLM call - Probability projections: domain-specific decay curves (h24/d7/d30) anchored to timeHorizon so probability equals projections[timeHorizon] - Data-driven cascade rules: moved from hardcoded array to JSON config (scripts/data/cascade-rules.json) with schema validation, named predicate evaluators, unknown key rejection, and fallback to defaults - 4 new cascade paths: infrastructure->supply_chain, infrastructure->market (both requiresSeverity:total), conflict->political, political->market - Proto: added Perspectives and Projections messages to Forecast - ForecastPanel: renders projections row and conditional perspectives toggle - 89 tests (19 new), all passing - Live-tested: OpenRouter produces perspectives from real data * feat(forecast): Phase 4 data utilization + entity graph Fixes data gaps that prevented 4 of 6 detectors from firing: - Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping - Chokepoint warm-ping (production-only, requires WM_API_BASE_URL) - Lowered CII conflict threshold from 70 to 60, gated on level=high|critical 4 new standalone detectors: - UCDP conflict zones (10+ events per country) - Cyber threat concentration (5+ threats per country) - GPS jamming in maritime shipping zones (5 regions) - Prediction markets as signals (60-90% probability markets) Entity-relationship graph (file-based, 38 nodes): - Countries, theaters, commodities, chokepoints, alliances - Alias table resolves both ISO codes and display names - Graph cascade discovery links predictions across entities Result: 51 predictions (up from 1-2), spanning conflict, infrastructure, and supply chain domains. 112 tests, all passing. * fix(forecast): redis cache format, signal source mapping, type safety Fresh-eyes audit fixes: - BUG: redisSet used wrong Upstash API format (POST body with {value,ex} instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes were silently failing, causing fresh LLM calls every run. - BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE, inflating confidence for market-derived predictions. - CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since generated Forecast type already has projections/perspectives fields. - CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE alerts when LLM calls add latency to seed runs. * feat(forecast): headline-entity matching with news corroboration signals Uses entity graph aliases to match headlines to predictions by country/theater (excludes commodity/infrastructure nodes to prevent false positives). Predictions with matching headlines get a news_corroboration signal visible in the panel. Also fixes buildUserPrompt to merge unique headlines from ALL predictions in the LLM batch (was only reading preds[0].newsContext). Live-tested: 13 of 51 predictions now have corroborating headlines (Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing. * feat(forecast): add country-codes.json for headline-entity matching 56 countries with ISO codes, full names, and scoring keywords (extracted from src/config/countries.ts + UCDP-relevant additions). Used by attachNewsContext for richer headline matching via getSearchTermsForRegion which combines country-codes + entity graph + keyword aliases. 14/57 predictions now have news corroboration (limited by headline coverage, not matching quality: only 8 headlines currently available). * feat(forecast): read 300 headlines from news digest instead of 8 Read news:digest:v1:full:en (300 headlines across 16 categories) instead of just news:insights:v1 topStories (8 headlines). Fallback to topStories if digest is unavailable. Result: news corroboration jumped from 25% to 64% (38/59 predictions). * fix(forecast): handle parenthetical country names in headline matching Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP region names before matching against country-codes.json. Also use includes() for reverse name lookup to catch partial matches. Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries with no current English-language news coverage. * fix(forecast): cache validated LLM output, add digest test, log cache errors Fresh-eyes audit fixes: - Combined LLM cache now stores only validated items (was caching raw unvalidated output, serving potentially invalid scenarios on cache hit) - redisSet logs warnings on failure (was silently swallowing all errors) - Added digest-based test for attachNewsContext (primary path was untested) - Fixed test arity: attachNewsContext(preds, news, digest) with 3 params * fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout - P2: Remove confidenceFromSources (dead code, computeConfidence overwrites all initial confidence values). Inline the formula in original detectors. - P3: Reduce warm-ping timeout from 30s to 15s (non-critical step) - P3: Add trial status comment on forecast panel config * fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push P1 fixes from code review: - CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran) via country-codes.json. Prevents substring false positives (IL matching Chile) in event correlation. Uses word-boundary regex for matching. - Market detector CII-to-theater mapping now uses entity graph traversal instead of broken theater-name substring matching. Iran correctly maps to Middle East theater via graph links. - Pre-push hook no longer runs destructive git checkout on proto freshness failure. Reports mismatch and exits without modifying worktree. * feat(forecast): add structured scenario pipeline and trace export * fix(forecast): hydrate bootstrap and trim generated drift * fix(forecast): keep required supply-chain contract updates * fix(ci): add forecasts to cache-keys registry and regenerate proto Add forecasts entry to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in cache-keys.ts to match api/bootstrap.js. Regenerate SupplyChain proto to fix duplicate TransitDayCount and add riskSummary/riskReportAction. * fix: restore desktop bootstrap timeouts and SW NetworkOnly (#1653) - Bootstrap: keep 1.2s/1.8s for web, restore 5s/8s for desktop (sidecar IPC + token resolution needs the headroom) - SW: revert navigation handler to NetworkOnly to prevent stale HTML caching that causes 404 storms on deploy * feat(supply-chain): restructure panel, add corridor disruption to shipping tab (#1652) * feat(supply-chain): restructure panel for scannable metrics and richer shipping tab Chokepoints tab: replace text blob description with structured metric rows showing warnings, vessels, WoW change, disruption %, and risk level. Move routing advisory into styled callout box on expand. Remove duplicated riskSummary and warning counts from server description field. Shipping Rates tab: add corridor disruption snapshot table showing per-corridor vessel counts, WoW traffic changes, disruption %, and risk levels. Tab now renders useful data even without FRED API key. Add CSS for expanded card state (previously no-op), metric rows, disruption color coding, routing advisory callout, and disruption table. * test(supply-chain): add restructure tests, update description assertion Add 7 structural tests for panel restructure: - activeHasData accepts chokepointData without FRED - renderShipping delegates to renderDisruptionSnapshot - null/empty loading states - data-cp-id and data-chart-cp preservation - conditional description rendering - server description no longer duplicates warning counts Update existing test: description now asserts counts are NOT in text (they moved to structured fields). * fix(seeds): lazy-load @aws-sdk/client-s3 in R2 storage helper (#1654) The top-level import crashes seed-forecasts on Railway when the package isn't installed. Dynamic import defers the load to when S3 mode is actually used, allowing the seed to run without the SDK when R2 is not configured. * test: cover runtime env guardrails (#1650) * fix(data): restore bootstrap and cache test coverage * test: cover runtime env guardrails * fix(test): align security header tests with current vercel.json Update catch-all source pattern, geolocation policy value, and picture-in-picture origins to match current production config. * debug(seeds): add R2 trace logging to seed-forecasts (#1655) Adds [Trace] log lines before/after R2 export and [R2] config resolution debug output to diagnose missing trace artifacts. * feat(search): add all missing panels to CMD+K command palette (#1656) 24 panels were not discoverable via CMD+K. Added commands for: map, webcams, AI insights/posture/forecasts, correlation panels (force posture, escalation, economic warfare, disaster cascade), fires, gulf economies, giving, UCDP events, displacement, climate, population exposure, security advisories, Israel sirens, telegram intel, airline intel, tech readiness, world clock, and layoffs. Coverage: 55/55 full-variant panels now searchable. * fix(health): close corridorrisk health monitoring gap (#1658) The corridorrisk raw key (2h TTL) expires between hourly seed cycles, causing health to report EMPTY even though data flows correctly through transit-summaries:v1. - Increase CORRIDOR_RISK_TTL from 2h to 4h (3 retries before expiry) - Add corridorrisk to ON_DEMAND_KEYS (WARN instead of CRIT when empty) * refactor: dedupe error mapper json responses (#1657) Extract jsonMessageResponse helper to replace 4 identical Response construction patterns in error-mapper.ts. * fix: use data-font attribute instead of inline style for font toggle (#1552) * fix: use data-font attribute instead of inline style for font toggle * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> * fix: use inline style for font-body-base to win cascade over variant themes Copilot review flagged that [data-font="system"] in @layer base loses to unlayered variant themes (e.g. happy-theme sets --font-body: Nunito). Switch back to inline style on --font-body-base so user font preference always wins the cascade, while keeping the RTL/CJK composition via var(--font-body-base) fallback. --------- Co-authored-by: Copilot Autofix powered by AI <[email protected]> Co-authored-by: Elie Habib <[email protected]> * feat(predictions): redesign panel with gradient bars, source accents, conviction labels (#1661) Visual redesign of prediction market cards: - Source badge (KALSHI/POLYMARKET) moved to card header with matching left accent stripe (blue for Kalshi, purple for Polymarket) - Gradient bars replace flat colors, with inner glow on strong signals - Conviction labels (Lean Yes / Lean No / Toss-up) based on threshold - Rounded bar corners (6px), thicker bars (28px), better spacing - Hover state on cards for interactivity feel * fix(predictions): prepend event title to short Kalshi outcome labels (#1659) Kalshi multi-outcome events return market titles like "Before 2035", "Rhode Island", "Johnny Depp" which are meaningless without the parent event context. Now combines event title with market title when the market title is short and doesn't contain a question mark. Before: "Before 2035" (KALSHI) After: "Will AGI be achieved?: Before 2035" (KALSHI) * fix(forecast): restore missing domains and split cyber (#1660) * fix(pwa): skip reload on first service worker registration (#1662) skipWaiting + clientsClaim fires controllerchange when the SW claims the page for the first time on a new session. The controllerchange listener was reloading unconditionally, causing a useless full page reload (double load of all assets and API calls) on every first visit. Now checks navigator.serviceWorker.controller at page load. If null (no prior SW), the controllerchange from initial claim is ignored. Deploy updates (new SW replacing existing) still trigger reload. * fix(health): prevent CF/CDN caching of health endpoint (#1665) CF was caching health responses despite no-cache, no-store because the Cache Rule marks /api/ as "eligible for cache" and CF interprets no-cache as "revalidate" not "don't cache". Changed to: - Cache-Control: private, no-store, max-age=0 (tells CF not cacheable) - CDN-Cache-Control: no-store (CF-specific override) * fix(health): add explicit seed-meta for chokepoints, move stale RPC keys to ON_DEMAND (#1667) - Chokepoints handler now writes seed-meta explicitly on fresh fetch (not on cache hits, per PR #1649 design) - Move riskScores and serviceStatuses to ON_DEMAND_KEYS (WARN not CRIT when stale; these are RPC-populated, not seeded) - cableHealth already in ON_DEMAND_KEYS * feat(trade): add US Treasury customs revenue to Trade Policy panel (#1663) * feat(trade): add US Treasury customs revenue to Trade Policy panel US customs duties revenue spiked 4-5x under Trump tariffs (from $7B/month to $27-31B/month) but the WTO tariff data only goes to 2024. Adds Treasury MTS data showing monthly customs revenue. - Add GetCustomsRevenue RPC (proto, handler, cache tier) - Add Treasury fetch to seed-supply-chain-trade.mjs (free API, no key) - Add Revenue tab to TradePolicyPanel with FYTD YoY comparison - Fix WTO gate: per-tab gating so Revenue works without WTO key - Wire bootstrap hydration, health, seed-health tracking * test(trade): add customs revenue feature tests 22 structural tests covering: - Handler: raw key mode, empty-cache behavior, correct Redis key - Seed: Treasury API URL, classification filter, timeout, row validation, amount conversion, sort order, seed-meta naming - Panel: WTO gate fix (per-tab not panel-wide), revenue tab defaults when WTO key missing, dynamic FYTD comparison - Client: no WTO feature gate, bootstrap hydration, type exports * fix(trade): align FYTD comparison by fiscal month count Prior FY comparison was filtering by calendar month, which compared 5 months of FY2026 (Oct-Feb) against only 2 months of FY2025 (Jan-Feb), inflating the YoY percentage. Now takes the first N months of the prior FY matching the current FY month count. * fix(trade): register treasury_revenue DataSourceId and localize revenue tab - Add treasury_revenue to DataSourceId union type so freshness tracking actually works (was silently ignored) - Register in data-freshness.ts source config + gap messages - Add i18n keys: revenue tab label, empty state, unavailable banner - Update infoTooltip to include Revenue tab description * fix(trade): complete revenue tab localization Use t() for all remaining hardcoded strings: footer source labels, FYTD summary headline, prior-year comparison, and table column headers. Wire the fytdLabel/vsPriorFy keys that were added but not used. * fix(test): update revenue source assertion for localized string * feat(supply-chain): add SCFI, CCFI, and BDI freight indices (#1666) * feat(supply-chain): add SCFI, CCFI, and BDI freight indices to shipping tab Transform the Shipping Rates tab from 2 lagging monthly FRED indices into a real-time freight cost dashboard with container and bulk shipping rates. Seed script: add fetchSCFI/fetchCCFI (SSE JSON API) and fetchBDI (HandyBulk HTML scrape) with inline history accumulation using source observation dates. Handler: make cache-only (seed is sole aggregator, no FRED fallback on miss). Panel: group indices into Container Rates, Bulk Shipping, Economic Indicators. Tests: 26 functional tests with fixture data for parsers, history, and handler. * fix(supply-chain): use raw Redis read and correct SCFI composite unit - Handler: switch from cachedFetchJson (env-prefixed) to getCachedJson(key, true) so preview deployments read the unprefixed seed key correctly - Seed: SCFI composite is a dimensionless index, not USD/TEU (route-level unit) - Tests: update assertions to match both fixes * fix(forecast): bump TTL from 3600s to 4800s to cover cron gap (#1668) Hourly cron + 3600s TTL = data expires right as the next seed starts, causing a ~30s EMPTY window. Bumped to 4800s (80min) so old data persists while the new seed runs. * fix(health): remove stale SEED_META for RPC-populated keys, bump to v2.6.5 (#1669) riskScores and serviceStatuses have data but permanently stale seed-meta (no longer written by cachedFetchJson after PR #1649). ON_DEMAND_KEYS only affects EMPTY status, not STALE_SEED. Removing their SEED_META entries so health doesn't check freshness for keys that can't update it. Also bumps version to 2.6.5. * fix(panels): refresh CMD+K active panels on open, add missing info tooltips (#1670) - Search: refresh activePanelIds in updateSearchIndex() so lazy-loaded panels (forecast, oref-sirens, telegram-intel, etc.) appear in CMD+K - Add ? info tooltips to AI Forecasts, Force Posture, and Disaster Cascade * fix(health): remove cableHealth SEED_META (RPC-populated, stale after PR #1649) (#1671) * fix(seed): bump BDI body size limit to 1MB, add gzip encoding (#1672) HandyBulk page is ~662KB, exceeding the 500KB guard. Also adds Accept-Encoding header to reduce transfer size. * fix(forecast): store full traces and tighten matching (#1673) * fix(forecast): restore missing domains and split cyber * fix(forecast): store full traces and tighten matching * fix(aviation): stop Vercel from calling AviationStack directly (#1674) * fix(aviation): stop Vercel from calling AviationStack directly - get-airport-ops-summary: read from relay seed cache (aviation:delays:intl:v3) instead of calling fetchAviationStackDelays() on every cache miss - list-airport-flights + get-flight-status: proxy through Railway relay /aviationstack endpoint instead of calling AviationStack from Vercel edge - Add /aviationstack proxy endpoint to ais-relay with 2min in-memory cache Vercel should NEVER call external paid APIs directly. Railway relay is the sole egress point for AviationStack (gold standard). * fix(config): update aviationStack feature to require WS_RELAY_URL Aviation handlers now proxy through Railway relay instead of calling AviationStack directly. Update runtime-config to reflect the actual dependency. * fix(supply-chain): force rebuild to deploy freight indices UI (#1680) Vercel build cache served stale bundle missing PR #1666 frontend changes (SCFI, CCFI, BDI grouping in SupplyChainPanel). Trivial whitespace change to invalidate Vite cache and force rebuild. * fix(trade): align flows cache key with seed (#1677) * fix(trade): align flows cache key with seed (US vs World, not China) The seed writes trade:flows:v1:840:000:10 (US vs World) but the data-loader requested trade:flows:v1:840:156:10 (US vs China), causing perpetual cache misses and a hidden Flows tab. * feat(seed): add bilateral trade flow pairs (US-China, US-Germany, etc.) Seed now writes both reporter-vs-World AND key bilateral pairs so switching between global and bilateral views hits warm cache. * feat(seed): add World-China and World-US bilateral flow pairs * fix(trade): revert flows to US-China (840/156), seed now covers this key The bilateral seed entries now write trade:flows:v1:840:156:10, so the original US-China request hits cache. Keeps the panel showing bilateral data consistent with the tariffs partner. * fix(forecast): log and honor full trace count (#1683) * fix(bootstrap): move shippingRates to FAST_KEYS for shorter CDN TTL (#1682) shippingRates was in SLOW_KEYS (2h CDN cache). After seed populates new freight indices (SCFI, CCFI, BDI), CF served stale bootstrap with only 2 FRED indices for up to 2.5h. Moving to FAST_KEYS (10min CDN TTL) ensures new data appears promptly. Also update BOOTSTRAP_TIERS in cache-keys.ts to match. * fix(handlers): pure Redis reads for 31 seed-backed Vercel handlers (#1684) * fix(handlers): convert 31 Vercel RPC handlers to pure Redis reads Enforce gold standard: Vercel reads Redis ONLY, Railway makes ALL external API calls. Replace cachedFetchJson (which fetches from external APIs on cache miss) with getCachedJson (pure Redis read from seed data). Converted domains: economic (8), market (7), trade (4), research (3), infrastructure (2), aviation (1), climate (1), cyber (1), natural (1), seismology (1), unrest (1), wildfire (1). Net -3,678 lines of external API call code removed from Vercel edge. * test: update structural tests for seed-only handlers Climate, wildfire, and unrest handlers no longer have TTL constants or ACLED imports since they are now pure cache readers. Update tests to verify getCachedJson usage instead of removed patterns. * fix(handlers): revert service-statuses, fix fred-batch key prefixing P1: Revert list-service-statuses.ts to original cachedFetchJson form. Both seeders (relay + cron) warm-ping the RPC which is the only writer. Making it read-only would break the data pipeline. This handler stays as-is until a proper seed script is built. P2: Replace getCachedJsonBatch with individual getCachedJson(key, true) calls in get-fred-series-batch.ts. getCachedJsonBatch uses prefixKey() which adds env prefix, but seed data is written without prefix. Using getCachedJson with raw=true bypasses the prefix. * fix(trade): prevent circuit breaker from caching empty results (#1685) All 5 trade fetch functions now pass shouldCache to skip caching when the RPC returns empty data. Previously, empty/unavailable responses were cached for 30 min in IndexedDB, causing Flows and Barriers tabs to stay hidden across page reloads even after the seed populated Redis with real data. * fix(supply-chain): move Economic Indicators to own tab (#1686) With 7 new freight index cards (SCFI, CCFI, BDI, BCI, BPI, BSI, BHSI) on the Shipping Rates tab, the FRED Economic Indicators were pushed too far down. Split into a 4th tab for better visibility. * fix(forecast): filter low-signal panel forecasts (#1687) * refactor: dedupe map URL param parsing helpers (#1675) Extract parseEnumParam and parseClampedFloatParam helpers in urlState.ts to replace 4 inline parsing blocks for view, zoom, lat, lon, and timeRange. * feat(trade): rename Revenue tab to US Revenue, add monthly bar chart (#1689) - Renamed "Revenue" tab to "US Revenue" for clarity - Added 12-month bar chart above the table showing customs revenue trend - Bars highlighted in red when revenue exceeds 1.5x prior year average - Makes the tariff impact (308% YoY increase) visually obvious * blog: add post on trade routes, chokepoints, and freight costs (#1692) Covers the new Supply Chain panel features: chokepoint monitoring (8 corridors), freight indices (SCFI, CCFI, BDI), trade policy (WTO data), and critical minerals. Opens with the current Hormuz crisis (94.4% traffic drop) as a real-world hook. * docs: update Mintlify docs for recent features (#1693) - finance-data.mdx: Add US Treasury customs revenue to Trade Policy section (Revenue tab, free API, FYTD comparison, desktop support) - features.mdx: Expand CMD+K section from 5 categories to full 55 panel coverage with organized subcategories - data-sources.mdx: Update advisory section for gold standard migration (Railway seed, 265-entry country map, relay proxy) - architecture.mdx: Add 12 missing seed scripts to Railway pipeline table (forecasts, conflict-intel, economy, supply-chain-trade, security-advisories, usni-fleet, gdelt-intel, research, etc.) - changelog.mdx + CHANGELOG.md: Add unreleased entries for customs revenue, advisory migration, CMD+K coverage, R2 traces, and 6 fixes * fix(ui): render heatmap with neutral 0% when sector change is null (#1691) Sectors with null change (market closed / bootstrap race) now render as 0% (neutral gray) instead of showing "temporarily unavailable". Error only shows when the data array is truly empty. * fix(gateway): skip CDN cache for upstreamUnavailable responses (#1690) When an RPC handler returns { upstreamUnavailable: true }, the gateway now sets Cache-Control: no-store instead of the normal CDN cache tier. This prevents CF from caching empty/error responses for hours (e.g. trade flows returning 0 results got cached for 4h with static tier). Successful responses continue to use their configured cache tier. * fix(blog): correct heroImage path and remove duplicate inline image (#1694) heroImage used /images/blog/ but blog base is /blog/, so the correct path is /blog/images/blog/. Also removed duplicate inline ![...] image reference since the hero image already shows it. * perf(news): skip client-side AI reclassification for digest items (#1695) * perf(news): skip client-side AI reclassification for digest items The server digest already runs enrichWithAiCache() which checks the same Redis keys that classifyEvent writes to. Re-firing classifyEvent from every client wastes edge requests even on cache hits. Removes the aiCandidates/classifyWithAI loop from the digest branch. Per-feed fallback path is unchanged (it never had classification). Saves up to ~48 classify-event edge requests per page load. * test(news): add regression test for digest branch no-reclassification guard 6 structural assertions ensuring digest-backed items never trigger client-side classifyWithAI calls, and that the imports are removed. Prevents accidental reintroduction of redundant edge requests. * fix(forecast): seed military surge signals (#1696) * fix(forecast): filter low-signal panel forecasts * fix(forecast): seed military surge signals * perf: gate non-critical startup loads and refresh intervals by viewport (#1697) * perf: gate non-critical startup loads and refresh intervals by viewport Scheduler (App.ts): - firms: gate by satellite-fires panel viewport or natural layer - intelligence: gate by cii/strategic-risk/strategic-posture viewport or country brief open - temporalBaseline: gate by same intelligence consumers - correlation-engine: gate by correlation panel viewport - strategic-posture: viewport check instead of panel-exists - strategic-risk: viewport check instead of panel-exists Startup (data-loader.ts): - giving: only load when panel near viewport - firms: only load when satellite-fires panel near viewport or natural layer active - intelligence: only load when cii/strategic panels near viewport - iranAttacks: only load when intelligence consumers near viewport Adds 3 centralized helpers: shouldRefreshIntelligence(), shouldRefreshFirms(), shouldRefreshCorrelation(). * fix(perf): add iranAttacks map layer guard, remove redundant natural check P2: iranAttacks startup now also loads when mapLayers.iranAttacks is enabled, preventing missing map pins on first load when the layer is on but strategic panels are off-screen. P3: Remove mapLayers.natural from shouldRefreshFirms() since natural events have their own independent scheduler entry. * fix(seeds): extend existing cache TTL on validation failure (#1705) When a seed fetches data but validation rejects it (e.g. FIRMS API returns 0 fires due to timeout), extend the existing key's TTL instead of letting it expire. Old data survives until the next successful fetch. Applies to all seeds using runSeed(). * refactor: dedupe upstash json reads across edge endpoints (#1708) * fix: unblock geolocation and fix stale CSP hash (#1709) * fix: unblock geolocation and fix stale CSP hash for SW nuke script Permissions-Policy had geolocation=() which blocked navigator.geolocation used by user-location.ts. Changed to geolocation=(self). CSP script-src had a stale SHA-256 hash (903UI9my...) that didn't match the current SW nuke script content. The script was silently blocked in production, preventing recovery from stale service workers after deploys. Replaced with the correct hash (4Z2xtr1B...) in both vercel.json and index.html meta tag. * test: update permissions-policy test for geolocation=(self) Move geolocation from "disabled" list to "delegated" assertions since it now allows self-origin access for user-location.ts. * fix(forecast): bundle military surge inputs (#1706) * chore: clear baseline lint debt (173 warnings → 49) (#1712) Mechanical fixes across 13 files: - isNaN() → Number.isNaN() (all values already numeric from parseFloat/parseInt) - let → const where never reassigned - Math.pow() → ** operator - Unnecessary continue in for loop - Useless string escape in test description - Missing parseInt radix parameter - Remove unused private class member (write-only counter) - Prefix unused function parameter with _ Config: suppress noImportantStyles (CSS !important is intentional) and useLiteralKeys (bracket notation used for computed/dynamic keys) in biome.json. Remaining 49 warnings are all noExcessiveCognitiveComplexity (already configured as warn, safe to address incrementally). * refactor(enrichment): dedupe domain/org normalization helpers (#1707) * refactor: dedupe github latest release fetch wiring (#1704) * fix(military): tighten flight classification (#1713) * fix(forecast): bundle military surge inputs * fix(military): tighten flight classification * refactor: dedupe edge api json response assembly (#1702) * refactor: dedupe edge api json response assembly * refactor: expand jsonResponse helper to all edge functions Roll out jsonResponse() from _json-response.js to 16 files (14 handlers + 2 shared helpers), eliminating 55 instances of the new Response(JSON.stringify(...)) boilerplate. Only exception: health.js uses JSON.stringify(body, null, indent) for pretty-print mode, which is incompatible with the helper signature. Replaced local jsonResponse/json() definitions in contact.js, register-interest.js, and cache-purge.js with the shared import. * fix(seeds): skip transient redis lock timeouts (#1714) * fix(seeds): skip transient redis lock timeouts * docs(seeds): clarify transient redis error matching * test: expand transient redis error coverage Add tests for ECONNRESET, DNS failure (EAI_AGAIN), ETIMEDOUT, and negative cases (HTTP 403, payload size) to confirm isTransientRedisError only matches network-level failures, not app-level Redis errors. * refactor(economic): dedupe FRED series fetch/parse helper (#1715) * fix(ucdp): page error logging, page-0 fallback, TTL extension (#1717) * fix(ucdp): add page error logging, page-0 fallback, and TTL extension on empty Three resilience improvements for UCDP seed loop: 1. Log actual error messages on page fetch failures instead of silently swallowing them. Enables diagnosing API outages vs rate limits. 2. Fall back to page 0 data when all newest-page fetches fail. Page 0 is already fetched during version discovery, so this is free. Provides partial (older) data instead of writing 0 events. 3. When 0 events remain after processing, extend existing Redis key TTL instead of overwriting with empty payload. Preserves stale-but-valid data for the next cycle rather than causing EMPTY_DATA CRIT in health. * fix(ucdp): remove page-0 fallback, stop seed-meta on failed fetches P1 fixes from review: - Remove page-0 fallback that overwrote last known good cache with stale historical data. Extend existing key TTL instead. - Stop writing fresh seed-meta timestamps when no new payload is written (both all-pages-failed and empty-after-filtering branches). Health checks should reflect actual data freshness, not failed attempts. Add 6 targeted source-analysis tests verifying: - Error logging on page failures - No page-0 data injection - TTL extension on failure branches - seed-meta only written on successful publish * fix(military): improve source-backed flight inference (#1716) * fix(military): improve source-backed flight inference * fix(military): tighten operator metadata matching * fix(military): tighten source hint inference * fix(blog): apply SEO and content improvements to supply chain post (#1728) - Shorten meta description to 155 chars (was truncating in SERPs) - Shorten metaTitle to under 60 chars - Add Key Takeaways box for skimmers and AI citation - Move brand name into first paragraph - Add "Data as of" timestamp note - Rename H2 with target keyword "maritime chokepoints" - Add 3 internal links to related blog posts - Add FAQ section (BDI, Hormuz oil impact, critical chokepoints) - Strengthen CTA with specific panel actions * feat(military): add enrichment audit waterfall (#1730) * fix(military): improve source-backed flight inference * fix(military): tighten operator metadata matching * fix(military): tighten source hint inference * feat(military): add enrichment audit waterfall * feat(military): capture live source shape gaps * fix(blog): SEO improvements across all 16 blog posts (#1729) - Trim metaTitles to under 58 chars (were 63-96, all truncated in SERPs) - Adjust meta descriptions to 150-160 char range - Add 2-3 internal blog post links per article for cross-linking - Add FAQ sections (2-3 questions each) for long-tail keyword capture - No content changes to article body text * fix(sidecar): block cloud fallback in Docker mode (#1726) Self-hosted Docker instances must not proxy unhandled routes to api.worldmonitor.app. When LOCAL_API_MODE=docker, cloudFallback is forced to false regardless of LOCAL_API_CLOUD_FALLBACK env var. Logs a warning if the user explicitly requested fallback. Prevents self-hosted users from unknowingly sending traffic to the production Vercel deployment. * fix(military): use opensky oauth in seed (#1738) * fix(military): use opensky oauth in seed * feat(military): log opensky source usage * fix(seeds): extend seed-meta TTL alongside data keys on fetch failure (#1724) When upstream APIs fail and seeds extend existing data key TTLs, the seed-meta key was left untouched. Health checks use seed-meta fetchedAt to determine staleness, so preserved data still triggered STALE_SEED warnings even though the data was valid. Now all TTL extension paths include the corresponding seed-meta key: - _seed-utils.mjs runSeed() (fetch failure + validation skip) - fetch-gpsjam.mjs (Wingbits 500 fallback) - seed-airport-delays.mjs (FAA fetch failure) - seed-military-flights.mjs (OpenSky fetch failure) - seed-service-statuses.mjs (RPC fetch failure) * feat(economic): add macro stress signals (clean cherry-pick) (#1719) Cherry-pick economic changes from codex/signal-architecture-phase1 onto clean main. Removed gitignored docs/internal/ file and 23 unrelated files from scope drift. Changes: - Add HY spread (BAMLH0A0HYM2), jobless claims (ICSA), 30Y mortgage (MORTGAGE30US), and GSCPI to ALLOWED_SERIES - Fix display math: toDisplayValue scales units correctly for WALCL (trillions) and basis-point series - Bump batch limit from 10 to 20 series - Preserve Redis-read-only pattern (no live FRED fetch on Vercel) * perf(sw): throttled event-driven SW updates (#1718) * perf(sw): replace 5-min polling with throttled event-driven updates Replace aggressive 5-minute registration.update() loop with throttled event-driven checks: on tab visibility, online reconnect, and an hourly visible-only fallback. Cross-tab throttling via shared localStorage timestamp prevents multiple tabs from checking simultaneously. Reduces /sw.js request volume from ~12/hr/tab to ~1/hr across all tabs while maintaining fast deploy pickup on tab return or reconnect. * fix(sw): stateful first-visit guard + failure-aware throttle Two fixes from review: 1. controllerchange guard: hadController was const, so first-visit tabs never reloaded on later deploys. Now let + set true after skipping the initial control-taking event. 2. Throttle writes timestamp after successful check (not before). On failure, retries every 5min instead of waiting the full 1hr cooldown. Uses separate localStorage keys for last-check vs last-success. * fix(sw): 15min poll interval + warn on update check failure - Change setInterval from 5min to 15min to reduce unnecessary ticks (throttle still gates to 1hr on success, 5min on failure) - Log update check failures with console.warn instead of silently swallowing them * fix(military): prefer direct opensky over proxy (#1742) * fix(forecast): improve ranking and enrichment coverage (#1745) * feat(forecast): add trace quality summary (#1746) * feat(forecast): add trace quality summary * fix(forecast): split traced and full-run quality metrics * refactor(military): dedupe Wingbits aircraft-details fetch helper (#1737) * refactor(military): dedupe Wingbits aircraft-details fetch helper * fix(military): correct cached details type in batch endpoint * refactor(api): dedupe in-memory IP rate limiter (#1740) * feat: add Radiation Watch with seeded anomaly intelligence, map layers, and country exposure (#1735) * feat(widgets): AI widget builder with live WorldMonitor data (#1732) * fix(docs): exclude /docs from CSP that blocks Mintlify (#1750) * fix(docs): exclude /docs from CSP header that blocks Mintlify scripts The catch-all /(.*) header rule applied Content-Security-Policy with SHA-based script-src to all routes including /docs/*. Mintlify generates dozens of inline scripts that don't match those hashes, causing 71 CSP errors and a completely blank docs page. Fix: change catch-all to /((?!docs).*) so /docs paths inherit only their own lightweight headers (nosniff, HSTS, referrer-policy). * fix(tests): update deploy-config test for docs CSP exclusion Test was looking for exact source '/(.*)', updated to match the new '/((?!docs).*)' pattern that excludes /docs from the strict CSP. * feat(sanctions): add OFAC sanctions pressure intelligence (#1739) * feat(sanctions): add OFAC sanctions pressure intelligence * fix(sanctions): strip _state from API response, fix code/name alignment, cap seed limit - trimResponse now destructures _state before spreading to prevent seed internals leaking to API clients during the atomicPublish→afterPublish window - buildLocationMap and extractPartyCountries now sort (code, name) as aligned pairs instead of calling uniqueSorted independently on each array; fixes code↔name mispairing for OFAC-specific codes like XC (Crimea) where alphabetic order of codes and names diverges - DEFAULT_RECENT_LIMIT reduced from 120 to 60 to match MAX_ITEMS_LIMIT so seeded entries beyond the handler cap are not written unnecessarily - Add tests/sanctions-pressure.test.mjs covering all three invariants * fix(sanctions): register sanctions:pressure:v1 in health.js BOOTSTRAP_KEYS and SEED_META Adds sanctionsPressure to health.js so the health endpoint monitors the seeded key for emptiness (CRIT) and freshness via seed-meta:sanctions:pressure (maxStaleMin: 720 matches 12h seed TTL). Without this, health was blind to stale or missing sanctions data. * feat(llm): support forecast model overrides (#1751) * refactor(sanctions): simplify handler to Redis-read-only, fix seed OOM risk (#1753) * refactor(sanctions): simplify handler to Redis-read-only, fix seed OOM risk Handler (424→56 lines): - Remove live OFAC fetch fallback from Vercel Edge handler: XMLParser, OFAC_SOURCES, fetchSource, collectPressure, cachedFetchJson fallback. Vercel reads Redis only; Railway makes all external API calls. - On seed miss/empty, return emptyResponse() matching the radiation pattern. Seed: - Fetch SDN then Consolidated sequentially instead of Promise.all. Combined parallel parse peaks at ~150MB, tight against 512MB heap limit. Tests: - Add gold standard compliance assertions (no XMLParser, no OFAC_SOURCES). - Add memory safety assertion (no Promise.all on OFAC sources). - Replace handler XML-function tests (removed code) with Redis-read assertions. * chore: exclude DMCA-TAKEDOWN-NOTICE.md from markdownlint * fix(csp): add missing Vercel inline script hash (#1756) Add sha256-903UI9my1I7mqHoiVeZSc56yd50YoRJTB2269QqL76w= to script-src to allow Vercel-injected inline script that was being blocked by CSP. * fix(sanctions): add fast-xml-parser to Railway scripts deps (#1755) seed-sanctions-pressure.mjs imports fast-xml-parser to parse OFAC SDN XML feeds, but the package was never added to scripts/package.json. Railway deploys crash with ERR_MODULE_NOT_FOUND on startup. * fix(widget): add @anthropic-ai/sdk to root deps for Railway (#1757) The relay dynamically imports @anthropic-ai/sdk but it was only in scripts/package.json. Railway Nixpacks only installs root deps, so the package was missing at runtime (ERR_MODULE_NOT_FOUND). * chore(forecast): log llm stage routing (#1754) * fix(sanctions): add progress logging to seed (fetch size, entry count, new count) (#1758) * fix(gdelt): make RPC handler Redis-read-only (gold standard) (#1759) * fix(relay): add root nixpacks.toml to install scripts/ deps on Railway (#1760) Railway builds from repo root (root_dir=""), so scripts/nixpacks.toml is skipped. Root packages are installed via npm ci but scripts/package.json deps (@anthropic-ai/sdk, fast-xml-parser, etc.) never land in /app/node_modules, causing ERR_MODULE_NOT_FOUND at runtime. Add a root nixpacks.toml that: - Preserves existing scripts/nixpacks.toml settings (curl, NODE_OPTIONS) - Adds `npm install --prefix scripts` to the build phase so all scripts/package.json deps are installed alongside root deps * fix(forecast): tighten quality and enrichment balance (#1761) * fix: enforce gold standard on UCDP and theater posture handlers (#1763) Make list-ucdp-events.ts Redis-read-only (was calling ucdpapi.pcr.uu.se directly from Vercel Edge). Reduces 160 lines to 28. Make get-theater-posture.ts Redis-read-only (was calling OpenSky + Wingbits directly from Vercel Edge). Reduces 273 lines to 38. Keeps live/stale/backup Redis fallback chain. Update theater posture tests to verify read-only behavior instead of testing upstream coalescing and cache writes. All UCDP/theater data assembly now exclusively on Railway (ais-relay.cjs + seed-military-flights.mjs). * test(sanctions): add 74 unit tests for seed parsing/transformation functions (#1762) Tests cover all pure helper functions in scripts/seed-sanctions-pressure.mjs: listify, textValue, buildEpoch, uniqueSorted, compactNote, extractDocumentedName, normalizeDateOfIssue, buildReferenceMaps, buildLocationMap, resolveEntityType, extractPartyName, extractPrograms, extractEffectiveAt, extractNote, sortEntries, buildCountryPressure, buildProgramPressure, and an end-to-end buildEntriesForDocument fixture. Uses node:vm to load pure functions in an isolated context, bypassing the loadEnvFile + runSeed side effects that fire on module import. * Add thermal escalation seeded service (#1747) * feat(thermal): add thermal escalation seeded service Cherry-picked from codex/thermal-escalation-phase1 and retargeted to main. Includes thermal escalation seed script, RPC handler, proto definitions, bootstrap/health/seed-health wiring, gateway cache tier, client service, and tests. * fix(thermal): wire data-loader, fix typing, recalculate summary Wire fetchThermalEscalations into data-loader.ts with panel forwarding, freshness tracking, and variant gating. Fix seed-health intervalMin from 90 to 180 to match 3h TTL. Replace 8 as-any casts with typed interface. Recalculate summary counts after maxItems slice. * fix(thermal): enforce maxItems on hydrated data + fix bootstrap keys Codex P2: hydration branch now slices clusters to maxItems before mapping, matching the RPC fallback behavior. Also add thermalEscalation to bootstrap.js BOOTSTRAP_CACHE_KEYS and SLOW_KEYS (was lost during conflict resolution). * fix(thermal): recalculate summary on sliced hydrated clusters When maxItems truncates the cluster array from bootstrap hydration, the summary was still using the original full-set counts. Now recalculates clusterCount, elevatedCount, spikeCount, etc. on the sliced array, matching the handler's behavior. * fix(forecast): address P2/P3 code review findings from PR #1761 (#1765) - Strip enrichmentMeta from bootstrap.js forecasts payload (seed-internal, not for clients) - Rename quietDomainBonus -> priorityDomainBonus (it applies to priority domains, not quiet ones) - Extract cyber score formula magic numbers into named constants (CYBER_SCORE_TYPE_MULTIPLIER, etc.) - Pre-compute analysisPriority in rankForecastsForAnalysis to avoid double-call per comparison - Log when filterPublishedForecasts weak-fallback gate suppresses forecasts - Log how many fallback narratives populateFallbackNarratives applies - Add // penalties comment header in computeAnalysisPriority * feat(ui): separate macro stress and energy complex panels (#1749) * feat(ui): separate macro stress and energy complex panels Cherry-picked from chained branch onto clean main. Splits the Economic panel into focused Macro Stress (FRED indicators) and Energy Complex (oil analytics + market tape) panels. Also fixes negative dollar-B changes now display with minus sign (was rendering as positive due to Math.abs without sign prefix). * fix: restore BIS/spending loading, split commodity/energy flags Addresses 2 blockers from PR review: 1. Restore BIS and USASpending data loading in all 3 call sites (prime tasks, periodic refresh, initial load). Restore full spending and centralBanks tab rendering in EconomicPanel while keeping the new macro stress indicators view. 2. Split shared commoditiesLoaded flag into separate metalsLoaded and energyLoaded flags so each panel falls back independently. Energy data arriving cannot suppress metals fallback fetch. EconomicPanel now has 3 tabs: Indicators (macro stress), Spending, Central Banks. Oil tab removed (lives in EnergyComplexPanel). * fix(i18n): restore economic panel translation keys for BIS/spending tabs PR removed i18n keys needed by the restored spending and centralBanks tabs (gov, centralBanks, awards, policyRate, etc.). Also updated the infoTooltip to accurately describe the mixed surface (macro + gov + central banks) instead of claiming macro-only. * fix(seeds): add empty-data guards and fix health semantics (#1767) Health semantics: - Add faaDelays + gpsjam to EMPTY_DATA_OK_KEYS (0 records = calm, not error) - Fix EMPTY_DATA_OK_KEYS branch to still check seed-meta freshness (prevents stale empty caches from staying green indefinitely) Seed guards: - seed-airport-delays: fix meta key in fetch-failure path (seed-meta:aviation:delays -> seed-meta:aviation:faa + seed-meta:aviation:notam) - seed-military-flights: add full TTL extension on zero-flights branch (was exiting without preserving any derived data TTLs) - seed-wb-indicators: add percentage-drop guard (new count < 50% of cached = likely partial API failure, extend TTL instead of overwriting) - ais-relay.cjs: same percentage-drop guard for WB dual writer Codex-reviewed plan (5 rounds, approved). * Add 7 new tier-1 countries to risk scoring and monitoring (#1768) * fix: include all curated countries in CII scoring (LB, IQ, AF, KR, EG, JP, QA) The CII was only scoring 24 "tier-1" countries while 7 curated countries (Lebanon, Iraq, Afghanistan, South Korea, Egypt, Japan, Qatar) had full configurations but were excluded from instability scoring. This expands TIER1_COUNTRIES to all 31 curated countries so every monitored country gets proper CII scores with baselines, keywords, bounding boxes, zone mappings, and travel advisory fallbacks. https://claude.ai/code/sessio…
nichm
pushed a commit
to nichm/worldmonitor-private
that referenced
this pull request
Jul 1, 2026
…oala73#1626) PR koala73#1596 removed the feeds but left the domains in the allowlist. The relay still accepted proxy requests for these 403-blocked domains from clients with cached old bundles. Removed: - breakingdefense.com (403) - www.arabnews.com (403) - www.aei.org (403) - mymodernmet.com (403) Updated all 3 copies: shared/, scripts/shared/, api/
koala73
added a commit
that referenced
this pull request
Jul 21, 2026
…t guards, fix relay www-tolerance (#5378) (#5399) * test(rss-proxy): wire test file into CI and lock the pre-fetch guards (#5378) api/rss-proxy.test.mjs was in neither test:data nor test:sidecar, so its 5 tests never ran in CI. Add it to test:sidecar and close the adversary-reachable coverage gaps the sweep found (5 -> 28 tests). The sweep flagged "SSRF hostname/userinfo confusion" as Critical. Probing the real predicate shows it is NOT a bypass: WHATWG new URL() strips userinfo into username/password, and isAllowedDomain only strips a leading "www." — so techcrunch.com.attacker.example, [email protected] and the trailing-dot FQDN form all resolve to a non-allowlisted hostname and 403. The real finding is that the guard had zero coverage; deleting it left the suite green while the handler fetched the attacker host. These tests close that. New coverage, each mutation-proven (14 mutations, every one killing exactly its target test and no others): - initial-host allowlist: suffix/userinfo/trailing-dot confusion + link-local metadata, asserting the attacker host is never fetched - auth: missing key -> 401, invalid key -> 401, both before any upstream call - rate limit: exhausted -> 429 with no feed fetch, plus the headroom case so the 429 is attributable to the limiter verdict, not to Upstash being set - protocol: file:// -> 400 (not the 403 domain verdict) - request shape: missing/malformed url -> 400, OPTIONS -> 204, non-GET -> 405, disallowed Origin -> 403 without echoing the attacker origin - response policy: relay-only routing + long cache TTLs, short TTLs on success, no CDN-Cache-Control on a failed upstream, non-2xx relay retry, content-type fallback, AbortError -> 504, and the Google News 20s vs default 12s deadline Drift fix surfaced by the new invariant test: RELAY_ONLY_DOMAINS still listed www.arabnews.com, which #1626 removed from the RSS allowlist as a dead feed domain. The allowlist runs first, so the entry could only ever 403 before the relay routing it exists for was consulted. Arab News is sourced via news.google.com, not a direct arabnews.com feed. RELAY_ONLY_DOMAINS is exposed via the repo's `export const __testing__ = {...}` test-only convention (matching api/health.js, api/bootstrap.js, api/mcp.ts) so the test can assert every relay-only host is also allowlisted, preventing the same drift from recurring — without widening the module's public surface. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * fix(rss-proxy): www-tolerant relay routing + collapse the drifted dev allowlist (#5378) Two fixes surfaced by the #5378 review, both approved for this PR. 1. Relay-only routing www/apex asymmetry (was: exact-match). `isAllowedDomain` is www-tolerant (strips/adds a leading `www.`) but the relay-only check was `RELAY_ONLY_DOMAINS.has(hostname)` — exact. Result: all 17 relay-only hosts were reachable via their alternate form (e.g. `cisa.gov` for the registered `www.cisa.gov`), which passes the allowlist but misses relay routing and gets direct-fetched from a Vercel edge IP these hosts block — paying the full 12s/20s timeout before the error fallback recovers. Dormant today (every registered feed uses the exact form), but structural. Fix: extract `hostMatchForms()` into api/_rss-allowed-domain-match.js and use it for BOTH the allowlist predicate and the relay-only check, so the invariant is structural rather than dependent on data-entry symmetry. New test (apex `cisa.gov` routes to the relay) is mutation-proven: reverting to `.has(hostname)` turns it red. 2. vite.config.ts held a THIRD copy of the RSS allowlist for the dev-server proxy that had drifted ~138 domains from prod (134 prod hosts 403'd in dev; 4 dev-only entries including the dead `www.arabnews.com`), while scripts/validate-rss-feeds.mjs claimed it was a kept-in-sync mirror. Replace the hand-maintained Set with a direct import of `isAllowedDomain` so dev and prod share one www-tolerant allowlist. Validator's mirror list drops 5 -> 4. Verified `vite build` succeeds with the edge-module import. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * test(rss-proxy): add negative-space + failure-branch coverage from review (#5378) Multi-lens review (security, correctness, adversarial, testing) confirmed the SSRF false-positive call and the [-1, 600] Upstash mock shape, but found the guard tests only rejected LOOKALIKES of allowlisted names, never a plain stranger — plus a few weak spots. 29 -> 33 tests, each new one mutation-proven. - Plain non-allowlisted stranger host rejected on BOTH the initial-host and the redirect-hop allowlist. Every prior negative case was a lookalike (techcrunch.com.attacker.example, [email protected], trailing-dot) or a raw IP, so loosening the guard to `!isAllowedDomain(h) && !h.endsWith('.com')` (admit any .com) stayed green. Now killed on both code paths. - Generic 502 'Failed to fetch feed' branch + its lone captureSilentError call site — previously untested — covered both reachable ways: a non-Abort direct-fetch throw with no relay, and a relay-only host with no relay. - Rate-limit headroom positive control given teeth: it returned 200 whether the limiter granted headroom OR threw and failed open. Now asserts the `[rate-limit] redis-error` degraded log never fired (proven: a garbage Upstash reply now fails the test instead of passing). - Google News deadline test's unbounded `while (!signal)` spin bounded + given a per-test timeout: a regression that stops the handler reaching fetch now fails in ~110ms with "handler never reached fetch" instead of hanging the CI runner forever (verified). - Guard-ordering test now fails all three early gates at once (bad Origin + no key + non-GET) so only ordering explains the 403 'Origin not allowed'. - CORS Allow-Methods pinned to exact 'GET, OPTIONS' (was substring /GET/); 429 now asserts it still carries CORS headers. Both mutation-proven. - AbortError test comment corrected to state the Sentry-suppression gate is NOT asserted (captureSilentError no-ops under NODE_TEST_CONTEXT), instead of implying coverage it lacks. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au
mitchross
added a commit
to mitchross/worldmonitor
that referenced
this pull request
Jul 22, 2026
* docs(solutions): capture two reusable learnings from the KV cutover (koala73#5338) (koala73#5383) * fix(mcp): make GET /mcp crawler-readable and fix the discovery cache key (koala73#5382) * fix(mcp): serve the human guide on GET /mcp and key discovery caches correctly A plain GET to /mcp returned the transport's spec-correct 405, which Google Search Console reports as "cannot access" — on www, on the apex, and on every variant subdomain. It now returns the mcp-server.md guide as text/markdown, and variant hosts 308 crawler GETs to the apex canonical. The discovery response is where the real hazard is. /mcp and /.well-known/mcp branch on Accept and Last-Event-ID, but the server card shipped `public, max-age=3600` with no Vary. Vercel's edge keys on URL alone, so a warmed discovery 200 was served back (x-vercel-cache: HIT) to a GET carrying `Accept: text/event-stream` — handing an MCP SDK client a JSON body where the transport contract requires 405. Reproduced on production against /.well-known/mcp before this change; that is koala73#4937's hang class arriving through the CDN instead of the handler. The card keeps its cacheability and gains Vary: Accept, Last-Event-ID. The transport URL goes further and stays no-store, so its correctness never depends on an intermediary honoring Vary. The variant 308 is built by hand rather than via Response.redirect() so it can carry Vary too — a 308 is cacheable by default (RFC 9110 15.4.9). Canonical stays apex per ARCHITECTURE.md:72 — /mcp is on the Cloudflare apex→www exemption list and the server card advertises the apex endpoint. POST and OPTIONS are never redirected (koala73#4938). mcp-live-smoke.mjs gains the probes that can actually see this: warm the cache with a plain GET, then fail if the SSE GET is anything but 405. Run against un-fixed production it reproduced all three defects independently. Claude-Session: https://claude.ai/code/session_01HHHP2TMJZUAYuzp6iGvYHD * docs(solutions): MCP crawler GET and the CDN discovery-cache replay Records the finding behind the /mcp fix: a cacheable discovery 200 on a URL that content-negotiates on request headers gets replayed by a URL-keyed edge cache to transport clients. Includes the production reproduction, the Vary-vs-no-store reasoning, and the landmine that the first version of the regression check (/\bAccept\b/i) passed against the un-fixed origin because `-` is a word boundary and it matched `accept-encoding`. Adds the Discovery Read vs. Transport Operation concept to CONCEPTS.md. Claude-Session: https://claude.ai/code/session_01HHHP2TMJZUAYuzp6iGvYHD * fix(mcp): harden discovery negotiation * fix(review): preserve MCP discovery contracts * fix(mcp): align HEAD discovery metadata * fix(deps): clear high-severity DoS advisories failing the security-audit gate (koala73#5395) * fix(analytics): swallow Umami beacon rejection leaking to Sentry (koala73#5393) * chore(lint): appease biome 2.4.9's promoted rules across scripts/ (unblocks all PRs) (koala73#5400) PR koala73#5395's lockfile refresh floated biome 2.4.7→2.4.9 inside the caret range; 2.4.9 promotes rules (useIndexOf, noAdjacentSpacesInRegex, noUselessContinue, noUselessStringRaw, useDefaultParameterLast, noUnusedFunctionParameters) that flag 20 pre-existing spots in scripts/. Main's path-filtered biome job hasn't re-linted scripts/ since, so every PR triggering a full lint now fails (first: koala73#5397). All fixes are behavior-neutral: indexOf/regex/String.raw rewrites are equivalence-preserving, unused params dropped, and selectTopStories keeps its maxCount=8 default under an explicit biome-ignore (the auto-fix would have silently changed the signature contract). * test(rss-proxy): wire test file into CI, lock the SSRF/auth/rate-limit guards, fix relay www-tolerance (koala73#5378) (koala73#5399) * test(rss-proxy): wire test file into CI and lock the pre-fetch guards (koala73#5378) api/rss-proxy.test.mjs was in neither test:data nor test:sidecar, so its 5 tests never ran in CI. Add it to test:sidecar and close the adversary-reachable coverage gaps the sweep found (5 -> 28 tests). The sweep flagged "SSRF hostname/userinfo confusion" as Critical. Probing the real predicate shows it is NOT a bypass: WHATWG new URL() strips userinfo into username/password, and isAllowedDomain only strips a leading "www." — so techcrunch.com.attacker.example, [email protected] and the trailing-dot FQDN form all resolve to a non-allowlisted hostname and 403. The real finding is that the guard had zero coverage; deleting it left the suite green while the handler fetched the attacker host. These tests close that. New coverage, each mutation-proven (14 mutations, every one killing exactly its target test and no others): - initial-host allowlist: suffix/userinfo/trailing-dot confusion + link-local metadata, asserting the attacker host is never fetched - auth: missing key -> 401, invalid key -> 401, both before any upstream call - rate limit: exhausted -> 429 with no feed fetch, plus the headroom case so the 429 is attributable to the limiter verdict, not to Upstash being set - protocol: file:// -> 400 (not the 403 domain verdict) - request shape: missing/malformed url -> 400, OPTIONS -> 204, non-GET -> 405, disallowed Origin -> 403 without echoing the attacker origin - response policy: relay-only routing + long cache TTLs, short TTLs on success, no CDN-Cache-Control on a failed upstream, non-2xx relay retry, content-type fallback, AbortError -> 504, and the Google News 20s vs default 12s deadline Drift fix surfaced by the new invariant test: RELAY_ONLY_DOMAINS still listed www.arabnews.com, which koala73#1626 removed from the RSS allowlist as a dead feed domain. The allowlist runs first, so the entry could only ever 403 before the relay routing it exists for was consulted. Arab News is sourced via news.google.com, not a direct arabnews.com feed. RELAY_ONLY_DOMAINS is exposed via the repo's `export const __testing__ = {...}` test-only convention (matching api/health.js, api/bootstrap.js, api/mcp.ts) so the test can assert every relay-only host is also allowlisted, preventing the same drift from recurring — without widening the module's public surface. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * fix(rss-proxy): www-tolerant relay routing + collapse the drifted dev allowlist (koala73#5378) Two fixes surfaced by the koala73#5378 review, both approved for this PR. 1. Relay-only routing www/apex asymmetry (was: exact-match). `isAllowedDomain` is www-tolerant (strips/adds a leading `www.`) but the relay-only check was `RELAY_ONLY_DOMAINS.has(hostname)` — exact. Result: all 17 relay-only hosts were reachable via their alternate form (e.g. `cisa.gov` for the registered `www.cisa.gov`), which passes the allowlist but misses relay routing and gets direct-fetched from a Vercel edge IP these hosts block — paying the full 12s/20s timeout before the error fallback recovers. Dormant today (every registered feed uses the exact form), but structural. Fix: extract `hostMatchForms()` into api/_rss-allowed-domain-match.js and use it for BOTH the allowlist predicate and the relay-only check, so the invariant is structural rather than dependent on data-entry symmetry. New test (apex `cisa.gov` routes to the relay) is mutation-proven: reverting to `.has(hostname)` turns it red. 2. vite.config.ts held a THIRD copy of the RSS allowlist for the dev-server proxy that had drifted ~138 domains from prod (134 prod hosts 403'd in dev; 4 dev-only entries including the dead `www.arabnews.com`), while scripts/validate-rss-feeds.mjs claimed it was a kept-in-sync mirror. Replace the hand-maintained Set with a direct import of `isAllowedDomain` so dev and prod share one www-tolerant allowlist. Validator's mirror list drops 5 -> 4. Verified `vite build` succeeds with the edge-module import. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * test(rss-proxy): add negative-space + failure-branch coverage from review (koala73#5378) Multi-lens review (security, correctness, adversarial, testing) confirmed the SSRF false-positive call and the [-1, 600] Upstash mock shape, but found the guard tests only rejected LOOKALIKES of allowlisted names, never a plain stranger — plus a few weak spots. 29 -> 33 tests, each new one mutation-proven. - Plain non-allowlisted stranger host rejected on BOTH the initial-host and the redirect-hop allowlist. Every prior negative case was a lookalike (techcrunch.com.attacker.example, [email protected], trailing-dot) or a raw IP, so loosening the guard to `!isAllowedDomain(h) && !h.endsWith('.com')` (admit any .com) stayed green. Now killed on both code paths. - Generic 502 'Failed to fetch feed' branch + its lone captureSilentError call site — previously untested — covered both reachable ways: a non-Abort direct-fetch throw with no relay, and a relay-only host with no relay. - Rate-limit headroom positive control given teeth: it returned 200 whether the limiter granted headroom OR threw and failed open. Now asserts the `[rate-limit] redis-error` degraded log never fired (proven: a garbage Upstash reply now fails the test instead of passing). - Google News deadline test's unbounded `while (!signal)` spin bounded + given a per-test timeout: a regression that stops the handler reaching fetch now fails in ~110ms with "handler never reached fetch" instead of hanging the CI runner forever (verified). - Guard-ordering test now fails all three early gates at once (bad Origin + no key + non-GET) so only ordering explains the 403 'Origin not allowed'. - CORS Allow-Methods pinned to exact 'GET, OPTIONS' (was substring /GET/); 429 now asserts it still carries CORS headers. Both mutation-proven. - AbortError test comment corrected to state the Sentry-suppression gate is NOT asserted (captureSilentError no-ops under NODE_TEST_CONTEXT), instead of implying coverage it lacks. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * test(pro): critical-path budget guard for the committed /pro build (koala73#5396) (koala73#5397) * test(pro): critical-path budget guard for the committed /pro build (koala73#5396) Any PR that rebuilds the pro app (a koala73#5374-class change) can silently regress the page's critical path; the only tripwire was the weekly DebugBear email, days later and averaged. This guard runs at PR time against the committed artifacts: 700KB critical-path budget (entry + modulepreloads + stylesheets; currently ~625KB), Clerk must never be referenced from the page HTML and must stay a dynamic import in the entry chunk (the 3MB parse behind the lab score of 63), and a 6MB whole-assets cap. Checkers are pure and teeth-tested against bad fixtures so the guard itself is proven able to fail. * fix(test): drop exports from the pro budget guard — biome noExportsInTest (error severity in 2.4.9) * fix(seed-research): isolate arXiv categories + retry + TTL so a single blip can't empty prod (koala73#5409) (koala73#5413) * fix(deps): clear fresh high advisories redding audit-lockfile on all PRs (koala73#5417) (koala73#5418) * feat(content): 13 blog posts — undocumented features (tenders, sanctions, aviation, radiation, energy dashboard…) + 'not Palantir' positioning (koala73#5403) * fix(pricing): call out 'License / API key included' on the API Starter tier (koala73#5419) * fix(rss-proxy): remove dead rsshub.app from allowlists (koala73#5414) * fix(deps): clear sharp + fast-xml-parser high advisories redding audit-lockfile on all PRs (koala73#5423) (koala73#5424) * fix(feed-digest): decode & last so RSS entities aren't decoded twice (koala73#5432) * fix(feed-digest): decode & last so RSS entities aren't decoded twice `decodeXmlEntities` decoded `&` first. That turns the escaped ampersand of `&lt;` into a live `&`, which the very next `.replace` immediately consumes as `<` — so a single pass decodes two levels. `&` has to go last. The visible damage differs by call site: - `extractTag` (titles): a headline whose literal text is `<script>` arrives escaped as `&lt;script&gt;` and comes back as `<script>` — raw markup injected into `ParsedItem.title` and emitted on the wire instead of the text the publisher wrote. - `extractDescription`: it strips tags *after* decoding, so the same input loses the words entirely — "XSS triggered by <script> tags in profile bios" becomes "XSS triggered by tags in profile bios". Also switch the numeric-reference branches from `String.fromCharCode` to `String.fromCodePoint`. `fromCharCode` truncates to 16 bits, so `😀` decoded to U+F600 (private use) instead of 😀. Out-of-range values are dropped rather than allowed to throw `RangeError`, which would fail the whole feed parse over one malformed reference. Adds `tests/news-feed-digest-entity-decode.test.mts`, including a produce-then-consume round-trip (escape a plain string, decode it, assert the original comes back). 5 of its 6 cases fail on the previous implementation. * test(feed-digest): cover the ' branch in the round-trip helper Review follow-up. The local escapeXml helper didn't escape apostrophes, so the round-trip assertion never produced ''' and never exercised that decode branch. Escapes it now, and adds an original containing apostrophes — escaping alone would not have reached the branch, since no existing case contained one. This case is coverage, not a second regression case: it round-trips on the old implementation too. The double-decode cases above it are what fail on main. --------- Co-authored-by: thejesh23 <[email protected]> Co-authored-by: Elie Habib <[email protected]> * fix(seed-utils): retry transient Redis blips on the read path + skip-path meta ops (koala73#5437) (koala73#5438) The gdelt-intel cache-merge fallback loads the previous canonical snapshot via verifySeedKey -> redisGet: 5s abort, no retry, and any non-OK status silently returned null. During the 2026-07-21 GDELT brownout one blip per run at the soft-budget boundary read as "no previous snapshot" - the merge no-op'd, validation failed, runs skipped without writing, and seed-meta aged 21h until the freshness gate fired, while the canonical key was perfectly healthy. Sibling unretried ops on the same skip path produced two exit-1 FATAL crashes (writeFreshnessMetadata SET abort). - redisGet: withRetry with the redisCommand tagging contract (permanent 4xx fail fast, 429 honors Retry-After, timeout/5xx backoff). External contract unchanged: HTTP failures still degrade to null (now loudly), thrown failures still propagate - both only after retries. - writeFreshnessMetadata: wrap the SET in withRetry so a single Upstash abort can't escape as FATAL on the validate-skip path. - readCanonicalEnvelopeMeta: retry before degrading - a blip here writes recordCount=0 with fetchedAt=NOW, resetting the freshness clock over real staleness. - seed-gdelt-intel _loadPrevious: replace the silent catch with a loud cache-merge warning so a dead fallback is visible in run logs. Tests: 13 new (red-first) covering retry/degrade/propagate contracts for all three helpers plus the seeder's default wiring; seeder suite 794 green. Closes koala73#5437 Claude-Session: https://claude.ai/code/session_01AKbRZXV6kKe8MTYpKZSLBM * Update blog post to 6 dashboards, fix variants count, and add Energy … (koala73#5408) * Update blog post to 6 dashboards, fix variants count, and add Energy Atlas section * Address PR review: fix panel count to 26, update stale 'five' reference, fix heading capitalization, and add trailing newline * review fixes: registry-true panel counts, energy deep-dive link, modifiedDate Panel counts refreshed against src/config/panels.ts (102/41/60/32/10/26 — four of the old numbers had drifted). Link the new energy post shipped in koala73#5403, pin pipeline claim to the 88 mapped in code, fix "All Six" casing, add modifiedDate + energy keyword. Claude-Session: https://claude.ai/code/session_01JEGono85MnwW7F9mm4rQEF --------- Co-authored-by: Elie Habib <[email protected]> * feat(content): receipts round — rewrite Palantir post, print the real scorecard, live radiation + tender records (koala73#5443) * docs(solutions): three-layer cache eviction runbook for the live product catalog (koala73#5422) * feat(blog): pinned-post support; pin the Palantir post to the top of the index (koala73#5446) --------- Co-authored-by: Elie Habib <[email protected]> Co-authored-by: Hamood Moin <[email protected]> Co-authored-by: Thejesh <[email protected]> Co-authored-by: thejesh23 <[email protected]> Co-authored-by: Claude <[email protected]>
mitchross
added a commit
to mitchross/worldmonitor
that referenced
this pull request
Jul 24, 2026
* docs(solutions): capture two reusable learnings from the KV cutover (#5338) (#5383) * fix(mcp): make GET /mcp crawler-readable and fix the discovery cache key (#5382) * fix(mcp): serve the human guide on GET /mcp and key discovery caches correctly A plain GET to /mcp returned the transport's spec-correct 405, which Google Search Console reports as "cannot access" — on www, on the apex, and on every variant subdomain. It now returns the mcp-server.md guide as text/markdown, and variant hosts 308 crawler GETs to the apex canonical. The discovery response is where the real hazard is. /mcp and /.well-known/mcp branch on Accept and Last-Event-ID, but the server card shipped `public, max-age=3600` with no Vary. Vercel's edge keys on URL alone, so a warmed discovery 200 was served back (x-vercel-cache: HIT) to a GET carrying `Accept: text/event-stream` — handing an MCP SDK client a JSON body where the transport contract requires 405. Reproduced on production against /.well-known/mcp before this change; that is #4937's hang class arriving through the CDN instead of the handler. The card keeps its cacheability and gains Vary: Accept, Last-Event-ID. The transport URL goes further and stays no-store, so its correctness never depends on an intermediary honoring Vary. The variant 308 is built by hand rather than via Response.redirect() so it can carry Vary too — a 308 is cacheable by default (RFC 9110 15.4.9). Canonical stays apex per ARCHITECTURE.md:72 — /mcp is on the Cloudflare apex→www exemption list and the server card advertises the apex endpoint. POST and OPTIONS are never redirected (#4938). mcp-live-smoke.mjs gains the probes that can actually see this: warm the cache with a plain GET, then fail if the SSE GET is anything but 405. Run against un-fixed production it reproduced all three defects independently. Claude-Session: https://claude.ai/code/session_01HHHP2TMJZUAYuzp6iGvYHD * docs(solutions): MCP crawler GET and the CDN discovery-cache replay Records the finding behind the /mcp fix: a cacheable discovery 200 on a URL that content-negotiates on request headers gets replayed by a URL-keyed edge cache to transport clients. Includes the production reproduction, the Vary-vs-no-store reasoning, and the landmine that the first version of the regression check (/\bAccept\b/i) passed against the un-fixed origin because `-` is a word boundary and it matched `accept-encoding`. Adds the Discovery Read vs. Transport Operation concept to CONCEPTS.md. Claude-Session: https://claude.ai/code/session_01HHHP2TMJZUAYuzp6iGvYHD * fix(mcp): harden discovery negotiation * fix(review): preserve MCP discovery contracts * fix(mcp): align HEAD discovery metadata * fix(deps): clear high-severity DoS advisories failing the security-audit gate (#5395) * fix(analytics): swallow Umami beacon rejection leaking to Sentry (#5393) * chore(lint): appease biome 2.4.9's promoted rules across scripts/ (unblocks all PRs) (#5400) PR #5395's lockfile refresh floated biome 2.4.7→2.4.9 inside the caret range; 2.4.9 promotes rules (useIndexOf, noAdjacentSpacesInRegex, noUselessContinue, noUselessStringRaw, useDefaultParameterLast, noUnusedFunctionParameters) that flag 20 pre-existing spots in scripts/. Main's path-filtered biome job hasn't re-linted scripts/ since, so every PR triggering a full lint now fails (first: #5397). All fixes are behavior-neutral: indexOf/regex/String.raw rewrites are equivalence-preserving, unused params dropped, and selectTopStories keeps its maxCount=8 default under an explicit biome-ignore (the auto-fix would have silently changed the signature contract). * test(rss-proxy): wire test file into CI, lock the SSRF/auth/rate-limit guards, fix relay www-tolerance (#5378) (#5399) * test(rss-proxy): wire test file into CI and lock the pre-fetch guards (#5378) api/rss-proxy.test.mjs was in neither test:data nor test:sidecar, so its 5 tests never ran in CI. Add it to test:sidecar and close the adversary-reachable coverage gaps the sweep found (5 -> 28 tests). The sweep flagged "SSRF hostname/userinfo confusion" as Critical. Probing the real predicate shows it is NOT a bypass: WHATWG new URL() strips userinfo into username/password, and isAllowedDomain only strips a leading "www." — so techcrunch.com.attacker.example, [email protected] and the trailing-dot FQDN form all resolve to a non-allowlisted hostname and 403. The real finding is that the guard had zero coverage; deleting it left the suite green while the handler fetched the attacker host. These tests close that. New coverage, each mutation-proven (14 mutations, every one killing exactly its target test and no others): - initial-host allowlist: suffix/userinfo/trailing-dot confusion + link-local metadata, asserting the attacker host is never fetched - auth: missing key -> 401, invalid key -> 401, both before any upstream call - rate limit: exhausted -> 429 with no feed fetch, plus the headroom case so the 429 is attributable to the limiter verdict, not to Upstash being set - protocol: file:// -> 400 (not the 403 domain verdict) - request shape: missing/malformed url -> 400, OPTIONS -> 204, non-GET -> 405, disallowed Origin -> 403 without echoing the attacker origin - response policy: relay-only routing + long cache TTLs, short TTLs on success, no CDN-Cache-Control on a failed upstream, non-2xx relay retry, content-type fallback, AbortError -> 504, and the Google News 20s vs default 12s deadline Drift fix surfaced by the new invariant test: RELAY_ONLY_DOMAINS still listed www.arabnews.com, which #1626 removed from the RSS allowlist as a dead feed domain. The allowlist runs first, so the entry could only ever 403 before the relay routing it exists for was consulted. Arab News is sourced via news.google.com, not a direct arabnews.com feed. RELAY_ONLY_DOMAINS is exposed via the repo's `export const __testing__ = {...}` test-only convention (matching api/health.js, api/bootstrap.js, api/mcp.ts) so the test can assert every relay-only host is also allowlisted, preventing the same drift from recurring — without widening the module's public surface. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * fix(rss-proxy): www-tolerant relay routing + collapse the drifted dev allowlist (#5378) Two fixes surfaced by the #5378 review, both approved for this PR. 1. Relay-only routing www/apex asymmetry (was: exact-match). `isAllowedDomain` is www-tolerant (strips/adds a leading `www.`) but the relay-only check was `RELAY_ONLY_DOMAINS.has(hostname)` — exact. Result: all 17 relay-only hosts were reachable via their alternate form (e.g. `cisa.gov` for the registered `www.cisa.gov`), which passes the allowlist but misses relay routing and gets direct-fetched from a Vercel edge IP these hosts block — paying the full 12s/20s timeout before the error fallback recovers. Dormant today (every registered feed uses the exact form), but structural. Fix: extract `hostMatchForms()` into api/_rss-allowed-domain-match.js and use it for BOTH the allowlist predicate and the relay-only check, so the invariant is structural rather than dependent on data-entry symmetry. New test (apex `cisa.gov` routes to the relay) is mutation-proven: reverting to `.has(hostname)` turns it red. 2. vite.config.ts held a THIRD copy of the RSS allowlist for the dev-server proxy that had drifted ~138 domains from prod (134 prod hosts 403'd in dev; 4 dev-only entries including the dead `www.arabnews.com`), while scripts/validate-rss-feeds.mjs claimed it was a kept-in-sync mirror. Replace the hand-maintained Set with a direct import of `isAllowedDomain` so dev and prod share one www-tolerant allowlist. Validator's mirror list drops 5 -> 4. Verified `vite build` succeeds with the edge-module import. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * test(rss-proxy): add negative-space + failure-branch coverage from review (#5378) Multi-lens review (security, correctness, adversarial, testing) confirmed the SSRF false-positive call and the [-1, 600] Upstash mock shape, but found the guard tests only rejected LOOKALIKES of allowlisted names, never a plain stranger — plus a few weak spots. 29 -> 33 tests, each new one mutation-proven. - Plain non-allowlisted stranger host rejected on BOTH the initial-host and the redirect-hop allowlist. Every prior negative case was a lookalike (techcrunch.com.attacker.example, [email protected], trailing-dot) or a raw IP, so loosening the guard to `!isAllowedDomain(h) && !h.endsWith('.com')` (admit any .com) stayed green. Now killed on both code paths. - Generic 502 'Failed to fetch feed' branch + its lone captureSilentError call site — previously untested — covered both reachable ways: a non-Abort direct-fetch throw with no relay, and a relay-only host with no relay. - Rate-limit headroom positive control given teeth: it returned 200 whether the limiter granted headroom OR threw and failed open. Now asserts the `[rate-limit] redis-error` degraded log never fired (proven: a garbage Upstash reply now fails the test instead of passing). - Google News deadline test's unbounded `while (!signal)` spin bounded + given a per-test timeout: a regression that stops the handler reaching fetch now fails in ~110ms with "handler never reached fetch" instead of hanging the CI runner forever (verified). - Guard-ordering test now fails all three early gates at once (bad Origin + no key + non-GET) so only ordering explains the 403 'Origin not allowed'. - CORS Allow-Methods pinned to exact 'GET, OPTIONS' (was substring /GET/); 429 now asserts it still carries CORS headers. Both mutation-proven. - AbortError test comment corrected to state the Sentry-suppression gate is NOT asserted (captureSilentError no-ops under NODE_TEST_CONTEXT), instead of implying coverage it lacks. Claude-Session: https://claude.ai/code/session_018RkPSPXZKPpBtvAhNZx3au * test(pro): critical-path budget guard for the committed /pro build (#5396) (#5397) * test(pro): critical-path budget guard for the committed /pro build (#5396) Any PR that rebuilds the pro app (a #5374-class change) can silently regress the page's critical path; the only tripwire was the weekly DebugBear email, days later and averaged. This guard runs at PR time against the committed artifacts: 700KB critical-path budget (entry + modulepreloads + stylesheets; currently ~625KB), Clerk must never be referenced from the page HTML and must stay a dynamic import in the entry chunk (the 3MB parse behind the lab score of 63), and a 6MB whole-assets cap. Checkers are pure and teeth-tested against bad fixtures so the guard itself is proven able to fail. * fix(test): drop exports from the pro budget guard — biome noExportsInTest (error severity in 2.4.9) * fix(seed-research): isolate arXiv categories + retry + TTL so a single blip can't empty prod (#5409) (#5413) * fix(deps): clear fresh high advisories redding audit-lockfile on all PRs (#5417) (#5418) * feat(content): 13 blog posts — undocumented features (tenders, sanctions, aviation, radiation, energy dashboard…) + 'not Palantir' positioning (#5403) * fix(pricing): call out 'License / API key included' on the API Starter tier (#5419) * fix(rss-proxy): remove dead rsshub.app from allowlists (#5414) * fix(deps): clear sharp + fast-xml-parser high advisories redding audit-lockfile on all PRs (#5423) (#5424) * fix(feed-digest): decode & last so RSS entities aren't decoded twice (#5432) * fix(feed-digest): decode & last so RSS entities aren't decoded twice `decodeXmlEntities` decoded `&` first. That turns the escaped ampersand of `&lt;` into a live `&`, which the very next `.replace` immediately consumes as `<` — so a single pass decodes two levels. `&` has to go last. The visible damage differs by call site: - `extractTag` (titles): a headline whose literal text is `<script>` arrives escaped as `&lt;script&gt;` and comes back as `<script>` — raw markup injected into `ParsedItem.title` and emitted on the wire instead of the text the publisher wrote. - `extractDescription`: it strips tags *after* decoding, so the same input loses the words entirely — "XSS triggered by <script> tags in profile bios" becomes "XSS triggered by tags in profile bios". Also switch the numeric-reference branches from `String.fromCharCode` to `String.fromCodePoint`. `fromCharCode` truncates to 16 bits, so `😀` decoded to U+F600 (private use) instead of 😀. Out-of-range values are dropped rather than allowed to throw `RangeError`, which would fail the whole feed parse over one malformed reference. Adds `tests/news-feed-digest-entity-decode.test.mts`, including a produce-then-consume round-trip (escape a plain string, decode it, assert the original comes back). 5 of its 6 cases fail on the previous implementation. * test(feed-digest): cover the ' branch in the round-trip helper Review follow-up. The local escapeXml helper didn't escape apostrophes, so the round-trip assertion never produced ''' and never exercised that decode branch. Escapes it now, and adds an original containing apostrophes — escaping alone would not have reached the branch, since no existing case contained one. This case is coverage, not a second regression case: it round-trips on the old implementation too. The double-decode cases above it are what fail on main. --------- Co-authored-by: thejesh23 <[email protected]> Co-authored-by: Elie Habib <[email protected]> * fix(seed-utils): retry transient Redis blips on the read path + skip-path meta ops (#5437) (#5438) The gdelt-intel cache-merge fallback loads the previous canonical snapshot via verifySeedKey -> redisGet: 5s abort, no retry, and any non-OK status silently returned null. During the 2026-07-21 GDELT brownout one blip per run at the soft-budget boundary read as "no previous snapshot" - the merge no-op'd, validation failed, runs skipped without writing, and seed-meta aged 21h until the freshness gate fired, while the canonical key was perfectly healthy. Sibling unretried ops on the same skip path produced two exit-1 FATAL crashes (writeFreshnessMetadata SET abort). - redisGet: withRetry with the redisCommand tagging contract (permanent 4xx fail fast, 429 honors Retry-After, timeout/5xx backoff). External contract unchanged: HTTP failures still degrade to null (now loudly), thrown failures still propagate - both only after retries. - writeFreshnessMetadata: wrap the SET in withRetry so a single Upstash abort can't escape as FATAL on the validate-skip path. - readCanonicalEnvelopeMeta: retry before degrading - a blip here writes recordCount=0 with fetchedAt=NOW, resetting the freshness clock over real staleness. - seed-gdelt-intel _loadPrevious: replace the silent catch with a loud cache-merge warning so a dead fallback is visible in run logs. Tests: 13 new (red-first) covering retry/degrade/propagate contracts for all three helpers plus the seeder's default wiring; seeder suite 794 green. Closes #5437 Claude-Session: https://claude.ai/code/session_01AKbRZXV6kKe8MTYpKZSLBM * Update blog post to 6 dashboards, fix variants count, and add Energy … (#5408) * Update blog post to 6 dashboards, fix variants count, and add Energy Atlas section * Address PR review: fix panel count to 26, update stale 'five' reference, fix heading capitalization, and add trailing newline * review fixes: registry-true panel counts, energy deep-dive link, modifiedDate Panel counts refreshed against src/config/panels.ts (102/41/60/32/10/26 — four of the old numbers had drifted). Link the new energy post shipped in #5403, pin pipeline claim to the 88 mapped in code, fix "All Six" casing, add modifiedDate + energy keyword. Claude-Session: https://claude.ai/code/session_01JEGono85MnwW7F9mm4rQEF --------- Co-authored-by: Elie Habib <[email protected]> * feat(content): receipts round — rewrite Palantir post, print the real scorecard, live radiation + tender records (#5443) * docs(solutions): three-layer cache eviction runbook for the live product catalog (#5422) * feat(blog): pinned-post support; pin the Palantir post to the top of the index (#5446) * fix(content): replace blog title cards with product imagery (#5452) * feat(agent-readiness): sandbox, docs-MCP JSON-RPC errors, schemamap, modular llms.txt (orank round) (#5469) * feat(agent-readiness): advertise SDKs in the agent view + homepage rel=alternate pointer (orank round 2) (#5476) * docs(blog): clarify WorldMonitor and Palantir positioning (#5480) * fix(payments): re-check Dodo before stale-subscription denial (#5447) * feat(blog): strengthen SEO and AI discoverability (#5475) * fix(payments): distinguish transient entitlement-lookup failure from confirmed denial (#5483) * ci(deploy-gate): retry the check-runs poll before posting a terminal 'pending' (#5479) (#5482) * fix(seed-gdelt-intel): degrade bookkeeping failures instead of crashing; brownout-scale timeline TTL; content-age opt-in (#5478) (#5481) * fix(auth): honor current API access during renewal checks (#5490) * feat(homepage): link Palantir positioning article (#5491) * ux(payments): billing-aware renewal/lapsed states instead of generic Upgrade CTA (#4771) (#5494) * feat(billing): pure billing UX state derivation for #4771 * feat(billing): expose renewalVerificationState from getSubscriptionForUser * feat(billing): billing-aware panel gating copy instead of generic Upgrade CTA * feat(billing): renewal-verification banner variants; gating follows subscription changes * fix(widget-agent): structured billing-verification denial before generic 403 * refactor(billing): localize banner via shared i18n keys; per-pass gating derivation; skip no-op CTA rebuilds * fix(review): cancelled-in-period coverage, billing-denial on-call log, behavioral + wiring test locks * docs: bump service-module count for billing-state.ts Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * docs: regenerate stats.json for billing-state service module Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * docs(solutions): compound #4771 learnings — coverage-semantics bug + i18n shell convention (#5495) Two learnings from PR #5494 plus a Billing & Entitlements vocabulary seed and an English Shell glossary entry in CONCEPTS.md. Claims grounding-validated against source and live GitHub state (28 claims, 2 corrected). Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * fix(notification-channels): bound convexRelay() with a 15s timeout (#5485) * fix(mcp): route sibling fetches through canonical API (#5517) * fix(python-sdk): avoid secret-scan literals (#5486) * fix: avoid Python SDK secret-scan literals * test(python-sdk): pin auth header contract --------- Co-authored-by: Elie Habib <[email protected]> * fix(auth): keep Pro brief denials out of session recovery (#5516) * fix(auth): keep Pro brief denials out of session recovery * Address PR review feedback (#5516) - Preserve successful premiumFetch route matches in the AST guard * fix(ci): update pro-test PostCSS lockfile Upgrade PostCSS past GHSA-6g55-p6wh-862q and refresh its Nanoid dependency. * fix(auth): close the #5379 adversarial sweep — resource exhaustion, state corruption, MCP entitlement gaps, and the inert live suite (#5385) * fix(auth): bound the Clerk plan lookup with AbortSignal.timeout (#5379) server/auth-session.ts lookupPlanFromClerk() fetched api.clerk.com with no timeout. validateBearerToken awaits it on every standard (non-template) session token — the ones without a `plan` claim — so a stalled Clerk let an authenticated caller pin gateway invocations open indefinitely. Adds signal: AbortSignal.timeout(3s), matching the budget already used by the other external auth lookups (server/_shared/user-api-key.ts and api/_user-api-key.js VALIDATION_TIMEOUT_MS). The existing catch already fail-softs, so a timeout degrades to 'free' exactly like an HTTP error. Regression test drives the real seam (signed RS256 JWT against a local JWKS server -> validateBearerToken -> lookupPlanFromClerk) with a never-settling Clerk stub, and additionally pins that a timed-out lookup does NOT poison the 5-minute plan cache with a 'free' verdict — the next request retries. Mutation-proved: removing the signal line turns the test red (actual: 'still-pending'). * fix(auth): validate user-API-key shape and canonical format before trusting it (#5379) Two gaps in server/_shared/user-api-key.ts, both on the live gateway auth path. 1. State corruption / EoP. cachedFetchJson<UserKeyResult> only CASTS its payload — a poisoned cache entry or upstream shape drift (e.g. {}) reached callers as a truthy 'authenticated principal' whose .userId read undefined. Adds isUserKeyResult(), an own-property runtime guard (hasOwnProperty, so a polluted Object.prototype.userId cannot authenticate a bare {}). null still passes through untouched as the legitimate negative-cache answer. The warn logs the type only — payload and key hash are credential material. 2. Malformed-key amplification. startsWith('wm_') let wm_x burn a SHA-256, a Redis round-trip and a Convex lookup per attempt. Now gated on /^wm_[a-f0-9]{40}$/ BEFORE hashing, matching the sibling api/_user-api-key.js. The regex is deliberately duplicated rather than imported (that module evaluates env at load and pulls redisPipeline + client-ip into the edge bundle for one regex); a test asserts the two literals stay byte-identical so drift fails CI. Also corrects the UserKeyResult interface, which was lying: it declared keyId/name required, but fetchFromConvex returns Convex's row verbatim ({id,userId,name}) so keyId is undefined on that path — only api/_user-api-key.js maps id->keyId. Requiring keyId in the guard would have 401'd every fresh Convex validation. Verified all three callers (server/gateway.ts, server/_shared/premium-check.ts, api/mcp/auth.ts) read ONLY .userId; api/mcp/types.ts already types the dep as {userId: string}|null. Mutation-proved: reverting the format guard reds 9 tests (including the 'backend never invoked' amplification assertions); neutering the shape guard reds 10. 28/28 green restored. * test(mcp): give the entitlement gate and both rate limiters teeth (#5379) Gaps 4, 9 and 10. api/mcp/auth.ts is unchanged — this is coverage only. Gap 4: checkMcpEntitlementGate enforces tier>=1, mcpAccess===true and validUntil>=now, but every existing test used ONE fixture violating all three at once, so any single predicate could be deleted with 144/144 still green. Replaced with a one-predicate-at-a-time matrix (each case violates exactly one predicate and satisfies the rest), plus strict-truthiness cases (mcpAccess: 'true' and 1 must still 401) and a getEntitlements-throws case. Covers both paths that reach the gate — pro and user_key — since user_key is the credential class that would otherwise silently skip it (#4859). Mutation-proved, re-verified independently by the orchestrator: drop tier<1 -> 4 tests red drop !mcpAccess -> 8 tests red drop validUntil<now-> 4 tests red drop !ent -> SURVIVES, and cannot be killed: the following lines read ent?.features?.tier ?? 0 etc., so a null ent already collapses to tier=0 and is rejected by tier<1. !ent is redundant defence-in-depth and is an equivalent mutant by construction — documented here rather than papered over with a test that fakes it. Gaps 9/10: applyPerMinuteLimit and applyAnonDiscoveryLimit were never exercised. Now pinned for all three limiters: absent limiter -> pass-through, under/over limit, the -32029 message text, emitMcpRateLimitHit payload, and the deliberate fail-OPEN on limiter throw. Bucket keys are asserted explicitly — key:<apiKey> for env_key, and pro AND user_key both mapping to pro-user:<userId>, which is the security-relevant claim that the two share one 60/min budget rather than stacking two. Anon discovery additionally pins the client-IP trust model: a spoofed x-forwarded-for cannot rotate the bucket, cf-connecting-ip is only trusted with matching CF_EDGE_PROOF_SECRET, and a missing IP falls back to a shared bucket rather than an empty key. 73 tests, all green. * test(auth): mutation-proof the bootstrap user-key cache, coalescing and timeout (#5379) Gaps 6, 7 and 8. api/_user-api-key.js is unchanged — this is coverage only. Gap 6: the cache-hit guard (truthy && object && typeof userId==='string' && userId.length>0) was correct but untested, so any conjunct could be deleted silently. The issue filed this as 'empty cached userId accepted'; that framing is wrong — the guard already rejects it. Each conjunct is now individually load-bearing: '' / 123 / null userId, and non-object hits (string, array, number, null) must all decline to trust the cache entry. Gap 7: request coalescing collapses N concurrent lookups of the same key hash onto ONE Convex round-trip. Deleting the coalesce() wrapper broke nothing, so a burst with one key could amplify 1:1 onto Convex. Now asserted three ways — 5 concurrent calls with the same key hit the backend exactly once and all resolve identically; 5 concurrent calls with DIFFERENT keys hit it 5 times (proving we measure coalescing, not caching); and a call after the first settles hits the backend again, proving the in-flight map's finally-delete cleanup does not leak entries. Gap 8: postConvexJson's AbortSignal.timeout(VALIDATION_TIMEOUT_MS) was unasserted, so its deletion would have reintroduced an unbounded auth fetch. Mutation-proved, re-verified independently by the orchestrator — each of the three deletions reds exactly one test: drop .length>0 -> 1 red drop coalesce() wrapper -> 1 red drop signal: AbortSignal... -> 1 red 38 tests, all green. * ci(auth): actually run the live cache/auth sweep, and fix the stale assertion it was hiding (#5379) Gap 5. tests/live-api-cache-auth-regression.test.mjs is wrapped in describe(..., { skip: !LIVE }) gated on LIVE_API_CACHE_TESTS=1, which nothing in the repo ever set — verified by grep across .github/, package.json and scripts/. The file is in the test:data glob, so every CI run 'passed' it while executing zero assertions; the issue's mutation proof (unconditional throw at the top of the describe) still exited 0. Confirmed locally: without the flag the runner reports 'tests 0, pass 0'. Adds a scheduled workflow modelled on the existing mcp-live-smoke.yml precedent — cron every 6h at :47 (offset from that job's :23 so two live probes don't hit prod from the same runner range in the same minute), push-to-main filtered to the suite + workflow, and workflow_dispatch. Not pull_request: the target is live production, so a PR run could neither exercise its own changes nor fail for reasons the PR caused. No npm ci — the suite imports only node:assert and node:test. No secrets provisioned; the one authenticated probe stays per-test gated on WM_LIVE_TEST_KEY and reports SKIP, not failure. Turning it on immediately surfaced a stale assertion, which is the whole point: the suite required mimeType 'application/json' for EVERY resources/list entry, with a comment asserting 'the catalog is now all concrete, metadata-only resources'. That stopped being true when the MCP-Apps ui:// fleet landed — production correctly serves ui://worldmonitor/country-risk.html as 'text/html;profile=mcp-app' (api/mcp/ui/shell.ts UI_RESOURCE_MIME_TYPE). Production is right; the never-executed test had rotted. Fixed by mirroring the rule the in-process sibling already uses (tests/mcp-resources.test.mjs:383): expected mimeType is chosen by URI scheme, so it remains an exact-match assertion — a resource declaring the WRONG one of the two still fails — and the payload is now verified to parse as what it declares (HTML doctype vs JSON.parse) rather than assuming JSON. Verified against production: 6 pass, 0 fail, 1 skip (the WM_LIVE_TEST_KEY case). Without the env var: still a clean 0-test skip. * test(security): catch identifier-vs-identifier secret comparisons (#5379) Gap 11. The #3803 timing-oracle guard's regex required the right operand to be process.env.* or a token starting with a bare `expected`/`EXPECTED`. Because `expected\b` has no word boundary inside `expectedSecret`, the guard sailed straight past the most natural way to write the bug: probeSecret !== expectedSecret The left arm had the mirror hole: `\b(?:secret)\b` never matched inside `probeSecret`, so even `probeSecret !== process.env.RELAY_SHARED_SECRET` leaked. Now matches a secret-bearing identifier compared against process.env.*, an expected* constant, or ANOTHER secret-bearing identifier, in either order, and supports member chains (`req.headers.token === expectedToken`). SECRET_VARS collapses to ['secret','token','bearer'] since matching moved from whole-word to substring — the compound entries are subsumed, and a generic 'key' is still excluded so cacheKey/sortKey don't drown the guard in noise. Structural change: the pattern and the comment-stripping step are extracted into exported helpers so the meta-test exercises the EXACT regex the real scan uses. The previous meta-test reconstructed a copy-pasted duplicate, so it could pass while the real scan diverged — a guard-testing-a-guard that proved nothing. The table is now the regex's contract, with must-NOT-match rows carrying equal weight: the correct timingSafeEqual idiom, presence/nullish checks, and type checks (typeof token === 'string') must never flag. Seven rows are REAL lines lifted from the api/ tree with file:line attribution (api/oauth/token.ts:725 grantType === 'refresh_token', api/notification-channels.ts:239, and the _mcp-grant-hmac token.length index compare) so the negative cases are grounded in code that actually exists rather than hypotheticals. False positives get guards deleted, which would return us to the original hole. Quoted literals are unreachable by construction: the operator is bracketed by \s*, never .*, so the matcher cannot step over a quote to reach a fragment inside a string. ALLOWLIST_FILES stays empty — the real api/ scan is green. Mutation-proved: dropping the ident-vs-ident arm reds the meta test. 3/3 green. * docs(auth): pin the entitlement-null fail-open posture where the code lives (#5379) The 'design risk' from the issue. Decision: KEEP fail-open. No behavior change — server/gateway.ts and entitlement-check.ts changes are comments only. The posture was previously articulated ONLY inside a test file, so a reader of server/gateway.ts saw a null-entitlement path serving 200 with no sign it was deliberate. The decision site now documents the posture, the blast-radius reasoning (fail-closed turns any Convex/Upstash blip into a fleet-wide 403 for every paying API customer at once), and what bounds the leak. Verified each bound rather than asserting it: - Not reachable by the never-subscribed: minting a wm_ key ITSELF requires an active entitlement with apiAccess (convex/apiKeys.ts throws API_ACCESS_REQUIRED otherwise), so no entitlement row ⇒ no key ⇒ this path is never entered. - Tier-gated routes do NOT inherit it: checkEntitlement fail-CLOSES on null. - Warm path re-resolves within the 15-min cache, so a lapsed user must sustain an outage rather than wait one out. entitlement-check.ts's docstring and catch comment claimed 'fail-closed … caller blocks the request', which is false for this caller and invited someone to 'fix' the fail-open as a bug. Both now state that null means UNRESOLVED and that the conclusion is caller-dependent, naming both callers. Also documents a MISCONFIGURATION HAZARD the original framing missed: a deploy missing CONVEX_SITE_URL or CONVEX_SERVER_SHARED_SECRET returns null for every user on every request, permanently. For the fail-open caller that is NOT a transient blip the cache heals — there is no warm path to recover to, so the gate is silently disabled indefinitely. Marked P1, not a degraded mode. Tests go 10 -> 19, all 10 originals preserved. New cases pin each boundary at its ACTUAL behavior, not an assumed one: null and undefined serve; {features:{}} and apiAccess:false 403 (a resolved-but-empty row fails CLOSED, so the hole is narrower than 'any malformed object'); {} and a throwing getEntitlements propagate rather than serve; and the warm path 403s once a downgrade resolves. Records one KNOWN GAP as a test rather than hiding it: an entitlement with apiAccess:true and a MISSING validUntil is served with expiry unchecked, since `undefined < Date.now()` is false. Not currently reachable — convex/schema.ts declares validUntil: v.number() as required — so this is latent robustness on the cache path, not a live hole. Filed for follow-up. Mutation-proved: flipping the null case to fail-closed reds 4 tests. 19/19 green. * test(auth): pin the Convex-misconfig null path that the fail-open bound assumes away (#5379) The gateway fail-open comment bounds its risk on 'the warm path re-resolves'. That premise assumes the entitlement EVENTUALLY resolves. A deploy missing CONVEX_SITE_URL or CONVEX_SERVER_SHARED_SECRET takes the early return, so every user resolves to null on every request with no self-healing path — the 15-min cache cannot warm what never resolves, silently disabling the #4611 apiAccess gate fleet-wide and indefinitely. Pinned here so the premise of that bound stays honest. Asserts repeated nulls across distinct userIds (not one coalesced in-flight promise) and a same-user retry (no recovery). Env is saved and restored in a finally, matching this file's existing convention — deleting without restoring would leak the broken env into any test appended after this one, which is the same module-state-leak class PR #5370 had to clean up. 20/20 green. * test(auth): make wm_ key fixtures canonical so they match what production can mint (#5379) Fallout from the validateUserApiKey format tightening two commits back, and a gap in my own blast-radius check: I cleared the consumers by grepping static imports, but server/gateway.ts reaches the validator through a DYNAMIC `await import('./_shared/user-api-key')`, so these suites exercised the real implementation rather than a mock and a static-import grep could not see it. Six tests across two files passed keys like 'wm_free_test_key' and 'wm_test_active_key' — readable placeholders that generateKey() (src/services/api-keys.ts) can never produce, since it always mints wm_ + 40 lowercase hex. They were asserting gateway behavior on an input production cannot generate. Replaced with canonical well-shaped fixtures behind named constants so the role stays legible at each call site. No assertion changed: the same x-user-id rewriting, entitlement gating and plan_key telemetry are still asserted, now with a realistic key. The Convex mocks in both files match on URL, not on the key or its hash, so the values are arbitrary provided they are well-shaped. This raises fidelity rather than accommodating the new guard — the old fixtures would have sailed past a format check that production has always effectively had at the Convex layer. tests/*.test.mts: 4352/4352 green (was 4346 pass / 6 fail). * fix(review): close the three P1s the review found in this PR's own work (#5379) Multi-persona review, including an independent cross-model adversarial pass (gpt-5.5 via Codex). Three P1s, two of them defects this PR introduced. 1. MISSING EXPIRY SERVED FOREVER (server/gateway.ts) — found independently by the security reviewer AND the cross-model pass, both P1/100, citing the same line. An entitlement with apiAccess:true and NO validUntil was SERVED: `undefined < Date.now()` is false, so the expiry arm silently no-opped. Worse than the documented null posture, which at least self-heals — this one re-resolves to the same shape every request, so a lapsed subscriber keeps the keyed surface permanently. The sibling MCP gate already got this right (`ent?.validUntil ?? 0`); the gateway now matches. The earlier commit shipped this as a pinned 'KNOWN GAP' test; that was the wrong call when the fix is one nullish-coalesce, so the test is flipped to assert 403 and a second test pins the narrower residual (a non-numeric validUntil still serves — the real fix there is runtime shape validation in getEntitlements, noted not hand-waved). 2. THE NEW CI GATE COULD PASS ON ZERO TESTS (.github/workflows/live-api-cache-auth.yml) — flagged by three independent reviewers. Setting LIVE_API_CACHE_TESTS was not enough: the suite is one `describe(..., { skip: !LIVE })` and `node --test` exits 0 when everything skips, so a rename or typo would have recreated the exact silent-green bug this workflow was written to fix. The suite's own `assert.equal(LIVE, true)` self-check cannot help — it is inside the skipped describe. The step now pins `--test-reporter=tap` (not the default, which Node selects by TTY-ness) and fails unless `# pass N` shows N>=1. Verified both ways: env var absent -> exit 1 with an actionable ::error::; present -> exit 0, 6 passing. 3. THE #3803 TIMING-ORACLE GUARD WAS PASSING VACUOUSLY (tests/no-non-timing-safe-secret-compare.test.mts). `stripComments` deleted 30.2% of api/ by bytes before the scan — measured and reproduced: a glob like `/*.openapi.json` inside a // comment reads as a block-comment OPENER and ate 4160 bytes of api/mcp/types.ts including `export interface RpcToolDef`, and // inside a URL literal truncated real lines. A violation in a swallowed region was invisible. The stripper is gone (raw scan finds zero false positives across all 174 files), and a new test plants a known violation into EVERY api/ file and requires the scan to flag every one — so any future normalisation that eats real code turns red instead of quietly passing. Also from the review: - Widened the same guard to two shapes it missed: a neutrally-named local vs a secret-NAMED env var (neither operand is secret-named, so every ident arm missed it), and an inline `headers.get('x-...-secret') !== expected` — the codebase's dominant header idiom, and the literal shape of #3803 itself. Negative rows pin that non-secret env vars and non-secret header keys stay unflagged. - server/auth-session.ts: added the AGENTS.md-mandated User-Agent to the Clerk fetch. Every other outbound server fetch sets one; this was the sole exception. - Replaced every cross-file numeric line citation in the new posture comments with symbol references. They were already wrong ON ARRIVAL — this PR's own added lines shifted them, so entitlement-check.ts:289 pointed at a function parameter and :229-233 at an unrelated Redis comment. A comment whose value is being checkable must not rot on commit. Verified: typecheck + typecheck:api clean; vitest 816/816; tests/*.test.mts 4353/4353; tests/*.test.mjs + cli 11719 pass / 0 fail; sidecar 230/230. Mutation-proved: removing `?? 0` reds the closed-gap test. * fix(review): restore silently-gutted coverage and make two failure modes observable (#5379) Second review round, from the reliability and testing reviewers. 1. SILENTLY GUTTED TEST (tests/mcp-proxy.test.mjs). Fallout from this PR's key format tightening, in a file my earlier blast-radius sweep missed precisely BECAUSE it kept passing. The test "rejects wm_ user keys when Convex validation cannot run" used the placeholder 'wm_user_abc123'; since the tightening that key is rejected at the format gate before hashing, so the test still returned 401 while covering none of the path its own comment claims to prove - including the MODULE_NOT_FOUND dynamic-import regression it was written to catch. A green test that stopped testing anything is the exact failure class this PR exists to fix. Now uses a canonically-shaped but never-minted key so the request reaches fetchFromConvex and is rejected there. 2. MY OWN COMMENT WAS FACTUALLY WRONG (server/_shared/entitlement-check.ts). The MISCONFIGURATION HAZARD note added earlier in this PR claims the state is "surfaced" by the one-time console.warn in getConvexSharedSecret(). That warn only fires when the SHARED SECRET is missing - a deploy missing only CONVEX_SITE_URL disabled the Convex fallback, and therefore the fail-OPEN #4611 apiAccess gate, with no signal whatsoever. Rather than weaken the comment to match the code, added getConvexSiteUrl() with its own one-time warn so the claim is true. One warn per variable is deliberate: warning on only one of the two is what created this hole. 3. SILENT PRO->FREE DEGRADATION (server/auth-session.ts). lookupPlanFromClerk's catch swallowed everything and returned 'free' with no logging on any path. The AbortSignal.timeout added earlier in this PR made that path newly reachable from an ordinary Clerk stall rather than only a hard network error, so a sustained Clerk outage would silently downgrade every PRO user to free and look identical to a fleet of genuinely free users. Now logs the reason. The verdict is still not cached, so the next request retries. Also found and filed, NOT fixed here: #5384 - server/_shared/user-api-key.ts cannot distinguish "key does not exist" from "Convex unavailable" and negative-caches both for 60s, so a transient 5xx 401s a paying customer for a full minute. Pre-existing; this PR's shape guard runs after cachedFetchJson and neither causes nor worsens it. Fixing it changes the negative-caching contract on a live auth path and deserves its own PR. Verified: typecheck clean; mcp-proxy + auth-resource-timeout 64/64; entitlement-check + gateway-user-key-apiaccess 40/40. * fix(review): de-flake the live sweep and pin the edge cache-key behavior it exposed (#5379) Third review round. A teammate reported the live suite failing against prod on an assertion my own run had passed, which turned out to be the most useful finding in the whole PR. ROOT CAUSE. The suite's fake-auth assertions target URLs the edge caches publicly for 600s, and the cache key does NOT include X-WorldMonitor-Key (`vary: Origin` only). Verified directly against production: cache-busted URL + invalid key -> x-vercel-cache: MISS -> 401, no-store same URL once warm + same key -> x-vercel-cache: HIT -> 200, public (age 39) So the ORIGIN auth logic is correct; the assertion outcome depended entirely on whether the CDN happened to be holding an anonymous response. My run passed and my teammate's failed for that reason alone. On a 6-hourly schedule that is an intermittently-red job no PR can fix - precisely how a guard earns its way into being ignored, which is the failure this PR exists to prevent. FIXES - Fake-auth probes are now cache-busted, so they test origin auth deterministically and keep testing the thing they are named after. Verified by three consecutive runs: 7 pass / 0 fail / 1 skip each time (previously the first assertion flipped with cache state). - Added an explicit test pinning the cached-anonymous behavior, so it is a stated property rather than a flaky side effect. It asserts STRUCTURALLY that the payload served to an invalid key is the anonymous envelope carrying only the requested public key - so an entitled payload landing in a public cache entry (the #4497 incident) goes red. It also passes cleanly if the behavior is later fixed to 401, taking the fail-closed branch, so fixing the underlying issue will not require touching the test. My first version of that assertion compared byte-lengths across two live requests and was itself flaky (61512 vs 61287) - weather data changes between requests and different edge nodes hold differently-sized entries. Replaced with the structural check; adding a flaky test to a commit about flakiness would have been a poor joke. Filed #5386 for the cache-key posture itself - the origin is right, the cache key needs a product decision (accept / add to Vary / bypass-on-header), and api/bootstrap-auth.test.mjs:302 currently asserts a contract production does not honor when warm. Not fixed here: it is a CDN-rule change, not a code change. ALSO FROM REVIEW - api/mcp/auth.ts: documented `!ent` as intentionally-redundant defence-in-depth. It is unkillable by mutation - every read optional-chains to a falsy default, so a falsy ent already forces tier=0/mcpAccess=false/validUntil=0 and is rejected three times over (verified across all six falsy values, and by a double mutation dropping `!ent` AND `tier < 1` which still 401s). Comment-only; git diff confirms no logic change. - server/auth-session.ts: corrected a comment that named VALIDATION_TIMEOUT_MS as living in server/_shared/user-api-key.ts. It does not - that file inlines the 3_000 literal; only api/_user-api-key.js has the constant. Verified: typecheck + typecheck:api clean; vitest 816/816; tests/*.test.mts 4353/4353; tests/*.test.mjs + cli 11719 pass / 0 fail; mcp-proxy 59/59; mcp-auth-entitlement-and-limits 73/73; live suite 7 pass / 0 fail / 1 skip, stable across three consecutive runs. * fix(review): make the anti-vacuous-pass test itself non-vacuous (#5379) Fourth review round, and the third layer of the same lesson. Two commits ago I removed stripComments because it deleted 30.2% of api/ and let the #3803 timing-oracle guard pass vacuously, and I added a companion test that plants a violation into every api/ file. Its comment promised: "any future normalisation step that eats real code turns this test red." That was false, and the reviewer proved it. The companion had its OWN private readFile+match loop and never touched the real scan's code path. I reproduced it: reintroduced the exact #3803-class stripper into the real scan's line ONLY, ran the suite, and all 4 tests passed — including the one whose entire purpose is catching that. A guard-of-a-guard with no teeth, which is precisely the bug it was written to prevent, one level up. FIX — route both paths through one seam. normaliseForScan() is now the single normalisation point between reading a file and matching it. It is the identity function today, deliberately; the point is that anything which ever transforms source MUST live there, because the real scan and the companion both call it. That shared routing is what turns the companion into an actual safety net. The companion also got two strengthenings, because sharing the seam alone was not sufficient: - A direct no-shrinkage assertion: normaliseForScan(source).length must equal source.length for every file. This states the violated property outright rather than waiting for a planted violation to happen to land in a swallowed region. - Violations are planted at THREE positions (top, middle, bottom), not just appended. A swallowing normaliser eats a REGION, not a whole file — appending only at the end survives a stripper that ate the middle, and the companion would have stayed green while the real scan was blind. My first version made exactly that mistake. MUTATION PROOF. Adding the old stripper to normaliseForScan now fails the companion, naming 127 affected files: normaliseForScan DROPPED source for 127 file(s): api/[...notfound].ts, api/_agent-metadata.ts, ... Anything the scan cannot see, it cannot flag — this is how the guard silently stops working. Before this commit the identical mutation left all 4 tests green. Also confirmed from the same review pass, no change needed: the live-sweep workflow's zero-assertion guard is sound. GitHub Actions runs `run:` steps under `bash --noprofile --norc -eo pipefail`, so errexit aborts the step on a genuine test failure before the grep is reached; the grep covers only the distinct "zero assertions ran" case, which is what it is for. tests/*.test.mts: 4353/4353 green. * docs(ci): register live-api-cache-auth.yml in both docs-stats gates (#5379) CI docs-stats went red on the new workflow. Adding a .github/workflows/*.yml file trips TWO separate gates, and passing the first locally does not imply the second: 1. npm run docs:check (docs-stats.mjs --check) -> needs a row in ARCHITECTURE.md's '## 11. CI/CD' table. Reproduced verbatim: 'ARCHITECTURE.md: CI workflow live-api-cache-auth.yml is not listed in the CI/CD table'. 2. The CI docs-stats job ALSO regenerates and freshness-checks docs/generated/stats.json, which stores workflowCount plus the workflow filename list. Ran npm run docs:stats and committed the result: workflowCount 21 -> 22, filename added. docs:check now reports OK, 80 doc claims match code. * fix(lint): drop exports from the secret-compare test (biome noExportsInTest) (#5379) CI biome went red with 3 errors, all mine: biome's lint/suspicious/noExportsInTest forbids exporting from a test file, and this branch had added three exports (buildSecretComparePattern, PATTERN_CASES, normaliseForScan). origin/main has zero exports in this file, so the branch introduced them. Removed the export keywords rather than suppressing the rule. The stated rationale for exporting was auditability from outside the runner, but nothing imports this file and the property that actually matters is unaffected: the real scan and the planted-violation companion are in the SAME module, so they still share one buildSecretComparePattern and one normaliseForScan seam. Re-verified the seam still has teeth after the change — adding a comment-stripper to normaliseForScan reds the companion, naming 124 files: normaliseForScan DROPPED source for 124 file(s): api/[...notfound].ts, ... 4/4 green restored; npm run lint (biome + enforce-safe-html) passes clean. * docs(solutions): capture the verify-the-verifier convention from the #5379 sweep New: docs/solutions/conventions/verify-the-verifier-mutation-test-every-detection-layer.md The durable lesson from #5379 / PR #5385 is not any individual auth bug — it is that four times in one PR, a layer built to DETECT silent failure had a silent failure of its own, and each was found only by breaking the detector and watching whether it noticed: 1. a live suite inert because nothing set its gating env var (CI passed it having run zero assertions); 2. the CI workflow written to fix that, which could itself pass on zero tests because node --test exits 0 when everything skips; 3. a security regression guard passing vacuously because its comment-stripper deleted 30.2% of api/ before matching; 4. the anti-vacuity companion added to fix (3), which had its own private code path and never exercised the real scan — proven by mutation. The doc's reusable core is the smell (a negative assertion passes silently when its input shrinks, so anything that can shrink the input must be separately pinned) and the three-part recipe, with the emphasis that part 1 alone is necessary but NOT sufficient — that is the layer-4 lesson and the part most likely to be skipped. Classified knowledge-track / convention rather than best-practice (the explicit fallback): this is a rule to apply on every future test, CI-gate, or guard PR. First doc in docs/solutions/conventions/. Overlap adjudicated as Moderate, not High, so created rather than merged: best-practices/test-guard-assertions-and-module-state-reset.md (PR #5369/#5370, two PRs earlier on this same auth surface) shares the mutation-proof PRINCIPLE but covers different mechanisms (JSON.stringify coercing Infinity; a leaked module-state reset). Cross-linked both directions in the Related section, along with the country-scope 'mirror test' doc and the ttl-staleness doc (whose static audit fails the opposite way — over-matching rather than under-matching). CONCEPTS.md: added a 'Test & Guard Verification' cluster defining Vacuous Guard and Mutation Proof — both used with precise project-specific meaning across five documented recurrences now, and neither previously defined. Grounded: every file:line citation verified against the tree; the central claim (real scan and companion both route through the shared normaliseForScan seam) confirmed at :268, :309 and :324; merge state confirmed OPEN/18-pass at write time. Frontmatter passes the parser-safety validator. * fix(review): harden auth verification guards * chore: refresh PR mergeability state * fix(auth): close review hardening gaps * test(auth): stub prevalidation limiter in gateway suite * test(mcp): wire pre-auth guard in world brief fixture * fix(global-tenders): spread SAM.gov request budget, stop retrying 429s (#5444) (#5457) * fix(global-tenders): spread SAM.gov request budget, stop retrying 429s (#5444) SAM.gov enforces a small per-key daily quota (10/day for non-federal keys). The hourly seed fetched SAM every tick and retried 429s in-run — up to ~72 requests/day against that budget — so once the quota tripped, every subsequent run 429'd, the source pinned at 'stale', and its age climbed past the 180-minute ceiling (health SEED_ERROR, empty US tender queries). - pace SAM fetches: skip the request while the previous success is fresher than 150 minutes (~9.6 requests/day), carrying the prior records through with lastSuccessfulAt untouched so staleness accounting stays honest - treat SAM 429s as non-retryable in-run via a retry429 opt-out in fetchResponse (quota-style limits do not clear in seconds; retries only burn more budget) - thread previousSnapshot through to source adapters so pacing can see the prior sam status Tests cover the pacing skip, interval expiry, unchanged first-run behaviour, and the no-retry-on-429 contract; all 28 existing tender tests unchanged. * fix(global-tenders): preserve paced source health (#5457) - keep stale and error SAM states degraded during paced runs - align the SAM health window with its effective request cadence - publish paced in the proto, clients, OpenAPI, and MCP contract * fix(security): update pro-test postcss (#5457) Pin postcss 8.5.12 to clear GHSA-6g55-p6wh-862q in the production dependency audit. --------- Co-authored-by: Elie Habib <[email protected]> * fix(forecast): extend EIA settlement grace (#5524) * fix(forecast): extend EIA settlement grace * test(forecast): cover missing EIA metric grace (#5524) * feat(api): enforce the sold daily cap + informative at-cap 429 (#4635) (#4684) The enforcement half of the #4635 lifecycle, behind API_RATE_LIMIT_ENFORCE (shadow-safe until the flip): - U5: reserveDailyMeter now flags at the SOLD allowance (Starter 1,000/day), not the 10x ceiling -- `overLimit: count > allowance` (was `count > allowance * CEILING_MULTIPLIER`). CEILING_MULTIPLIER removed; the `allowance <= 0` guard keeps Enterprise (-1) uncapped. Renamed the meter's `overCeiling` field -> `overLimit`; the daily X-RateLimit-Limit header now advertises the sold cap (was 10x). - U4: the burst + daily 429 bodies now carry { error, plan, limit, limit_type, reset, upgrade_url } instead of a bare "Too many requests" / "Daily request ceiling exceeded". `planKey` is hoisted at the 429 sites; enterprise (top tier) omits upgrade_url. Additive body contract; X-RateLimit-* header shape unchanged. Telemetry reason strings (rl_ceiling_*) kept for dashboard/audit continuity. Meter 16/16 + gateway 18/18 tests green; typecheck:api clean. Unblocks the API_RATE_LIMIT_ENFORCE flip once the notify/upgrade stack lands. Claude-Session: https://claude.ai/code/session_01SjVX3GyMBmC2XyFWCHogN1 * fix(security): bump find-my-way to 9.7.0 — GHSA-c96f-x56v-gq3h (HTTP/2 DDoS) (#5527) A new high advisory against find-my-way <= 9.6.0 (fastify's router, transitive in consumer-prices-core) landed today and reds the audit-lockfile (consumer-prices-core) check on every PR, while path-filtered main stays green. Lockfile-only bump to the first patched version (9.7.0); verified with the exact CI invocation: node .github/scripts/audit-production-dependencies.mjs --workspace consumer-prices-core ... -> "Production audit OK ... 0 high+ advisories" Unblocks #5526 and any other open PR. Claude-Session: https://claude.ai/code/session_01StNurp4TGC3JLHbTtKJhbp * fix(payments): restore lapsed subscriber reactivation path (#5532) * fix(payments): restore lapsed subscriber reactivation path * fix(deps): patch vulnerable find-my-way release * fix(ui): cancel stale pro banner removal * fix(review): correct subscriber reactivation edge cases * chore(bootstrap): wind down the R2-origin experiment (DebugBear sampling + KTD7 no-go) (#5536) * fix(seo): remove unsupported schemamap robots directive (#5544) * test(bootstrap): cover DebugBear sampling boundaries (#5545) * Merge commit from fork * fix(payments): persist failed Dodo webhook incidents (#5533) * fix(payments): persist failed Dodo webhook incidents * fix(payments): close Dodo webhook failure lifecycle * Address PR review feedback (#5533) - Keep recovery bookkeeping errors out of processing incident recording\n- Serialize failure lifecycle updates with the seeded aggregate lock\n- Add concurrent regression coverage and deploy seeding\n\nNote: pre-existing failure in convex/__tests__/poolSelection.test.ts not addressed by this PR. * fix(payments): harden webhook failure rollout * fix(review): dedupe resolution transition, cover aggregate lifecycle branches - Extract applyWebhookFailureResolution so manual resolve and provider-retry recovery share one timestamped transition (review: maintainability) - Add lifecycle tests for changed-eventType redelivery and record-after- resolve reopening (review: testing) - Correct the ops-signal comment: production attempts the queue and logs scheduler failures without changing the retry response (review: maintainability) * fix(giving): disclose published benchmark provenance (#5540) * fix(giving): disclose published estimate provenance * fix(giving): disclose benchmark provenance in panel * test(giving): verify benchmark provenance in browser * fix(review): harden giving provenance states * fix(ci): refresh documentation component counts * fix(ci): retain valid giving data on refresh failure * fix(giving): close review recovery gaps * fix(payments): dead-letter authenticated malformed webhooks instead of 401 (#5547) Split signature verification from payload validation in the Dodo webhook handler. 401 is now reserved for credentials that fail HMAC verification; a well-signed payload that fails JSON parsing or schema validation records a sanitized dead-letter projection and returns 500, so permanent provider- side defects exhaust retries into a repairable incident rather than a mislabeled signature failure (review: adversarial, PR#5533 finding #1). - verifyDodoSignature mirrors the SDK's vendored standardwebhooks scheme (whsec_ base64 secret, HMAC-SHA256, v1 signatures, 5min tolerance) - Payload validation uses the SDK's exported WebhookPayloadSchema, so the accepted shape is identical to verifyWebhookPayload - Extract persistFailureAndSignal shared by the validation and processing failure paths - Tests: signed schema-invalid and unparseable bodies dead-letter with 500; a bad signature still 401s with no failure row * fix(routing): redirect pricing to canonical section (#5548) * feat(i18n): Hoàn thiện bản dịch tiếng Việt (vi.json) (#5539) * feat: hoàn thiện bản dịch tiếng Việt cho vi.json Dịch 168 giá trị chưa dịch sang tiếng Việt (từ 247 giá trị ban đầu). 79 giá trị còn lại giữ nguyên (thuật ngữ kỹ thuật/tên riêng: PRO, AI, DDoS, WTO, AWACS, macOS, WhatsApp, MMSI, SQUAWK...). Các danh mục đã dịch: - header: Tech News → Tin tức Công nghệ, Cybersecurity → An ninh mạng... - panels: Gold Intelligence → Thông tin Vàng, Fear & Greed → Sợ hãi & Tham lam... - components: Central Banks → Ngân hàng Trung ương, Data Freshness → Độ mới Dữ liệu... - popups: US/NATO → MỸ/NATO, CHINA → TRUNG QUỐC, Recent Tracking → Theo dõi gần đây... - modals: Add → Thêm, download banners... - dashboardTabs: Main → Chính, New Tab → Tab mới, Add tab → Thêm tab... - countryBrief, commands, widgets, preferences, premium * fix(i18n): correct Vietnamese tooltip translations --------- Co-authored-by: Elie Habib <[email protected]> * fix(docker): preserve caller auth for local MCP requests (#5492) * fix(docker): preserve caller auth for local MCP requests Fixes #5471 Signed-off-by: Arpit Tagade <[email protected]> * fix(sidecar): strip transport token from cloud proxies --------- Signed-off-by: Arpit Tagade <[email protected]> Co-authored-by: Elie Habib <[email protected]> * fix(docker): preserve caller auth for local MCP requests (#5537) * fix(docker): preserve caller auth for local MCP requests Fixes #5471 Signed-off-by: Arpit Tagade <[email protected]> * fix(sidecar): strip transport token from cloud proxies --------- Signed-off-by: Arpit Tagade <[email protected]> Co-authored-by: Elie Habib <[email protected]> * fix(seo): prevent raw-text crawlers from extracting "WWorld Monitor" Move the decorative "W" brand mark from inline text to a CSS ::after pseudo-element. Raw textContent extraction now yields "World Monitor" instead of "WWorld Monitor". The visual appearance, accessible name, and first-paint footprint are unchanged — the ::after inherits the same font-size and color from .skeleton-brand-mark. Adds focused regression tests in deploy-config.test.mjs asserting: - Raw text does not contain "WWorld" - The brand mark span has no text content - The "W" is rendered via CSS content Fixes #5541 (#5549) * feat(activation): day-0 pro activation onboarding interstitial (#5534) * feat(activation): pure pro-activation state core — mount decision, step model, fire-once keying (U1) Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * feat(activation): interstitial shell — overlay, step chrome, focus trap, exit summary, en copy (U3) Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * feat(activation): brief/alerts/power step wiring + finish-setup chip (U4-U6) Brief: atomic setNotificationConfig with explicit hour+IANA tz, insights world-brief preview, inline hour select. Alerts: pre-denied blocked state, patch-not-clobber channels, cadence-honest copy. Power: injected deep links + R8 settings pointer. Chip: versioned-key dismissal. Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * feat(activation): checkout-return marker + post-reload boot mount hook (U2) Marker written before clearCheckoutAttempt on the success branch only; mount decision evaluated off the boot critical path with bounded snapshot-retry. Surfaces subscriptionId/currentPeriodStart on getSubscriptionForUser (additive, existing columns) for the fire-once key — accepted plan deviation. Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * fix(activation): re-arm mount retry on subscription snapshot changes too With the real subscription snapshot as the fire-once key input, a boot with live entitlement but a not-yet-loaded subscription snapshot would stall in 'keep' forever watching entitlement only. Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * feat(activation): proActivation locale fan-out — 24 locales, register-calibrated (fa per its file convention) Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * feat(activation): funnel telemetry + end-to-end spec (U7) Typed Umami events with whitelisted minimized payloads (planKey/step/exit counts — never billing identifiers); single entered fire site at mount; 7 Playwright scenarios green against the dev server. Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * refactor(activation): simplify pass — shared focus-trap util, leaf record parsers, type reuse, idle-handle cleanup Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * fix(review): apply findings #1-4, #8-12, #14 — hour-capture ref, seeded alerts fallback, preview tri-state, coverage locks P0 #1: digest hour captured in a closure ref so the in-flight re-render can't wipe the user's pick. P1 #2: alerts catch-path seeds from flow context (+email when brief confirmed) instead of empty — convex channels field is full-replace. P1 #3 + P2 #8-12: failed-state e2e, chip assertion, expired-Pro branch, payload combo, catalog-derived drift guard, focus-trap unit tests. P2 #4 tri-state preview guard. P3 #14 unexported helper. Claude-Session: https://claude.ai/code/session_01VUcnpsWficDVsPmEZEUJP7 * fix(review): apply findings #6, #7, #13 — account-scoped records, cross-tab mount claim Marker/fire-once/chip records carry the Clerk userId; foreign-user markers never mount (and are left for the buyer, TTL-reaped); unscoped marke…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #1596 removed feeds but left domains in the allowlist. The relay still proxied requests for these 403-blocked domains from clients with cached old bundles.
Removed from all 3 copies (shared/, scripts/shared/, api/):
Relay will now reject proxy requests for these domains.