feat: Gulf Stock Market & Economy Tracker for all 6 GCC Countries#641
feat: Gulf Stock Market & Economy Tracker for all 6 GCC Countries#641aa5064 wants to merge 670 commits into
Conversation
- Re-enable the ML worker import that was disabled - Remove @xenova/transformers from rollup externals so it gets bundled - Previous config had MLWorkerClass set to null, causing "not a constructor" error
- Show "Initializing ML models..." when worker not ready - Show "Analyzing sentiment..." during sentiment classification - Show "Generating world brief..." during summarization - Add animated spinner and pulsing text for visual feedback
- Add /api/groq-summarize.js - Llama 3.3 70B (1000 req/day free) - Add /api/openrouter-summarize.js - Fallback (50 req/day free) - Create summarization service with fallback: Groq -> OpenRouter -> Browser T5 - Browser T5 only loads when cloud APIs fail (lazy loading) - Add detailed progress bar with Step X/Y indicator - Show provider badge (groq/openrouter) on World Brief - Cache summaries for 30 min to respect rate limits - Cooldown increased to 2 min between brief generations Required env vars for Vercel: - GROQ_API_KEY (get from console.groq.com) - OPENROUTER_API_KEY (get from openrouter.ai)
- Add Upstash Redis caching to groq-summarize.js and openrouter-summarize.js - Switch to llama-3.1-8b-instant (14.4K/day vs 1K for 70b) - Cross-user cache deduplication with 24h TTL - Remove client-side cache (server handles all caching now) Improve AI Insights story selection: - Composite importance score: sources × velocity × recency + alert bonus - Source diversity cap: max 3 stories from same source - Recency decay: newer stories rank higher (12h half-life) Scalability: ~144x headroom (was hitting 1K/day limit)
…lusions - Remove 'breaking', 'emergency' from alert keywords (too generic) - Add ALERT_EXCLUSIONS for lifestyle/entertainment content - Remove confusing 'Major' tier label (was showing for T2 sources) - Tier badge now just shows dot for T2, 'Wire' for T1
World Brief in AI Insights already provides panel-level summarization. Also reduced ML batch size and timeout to prevent embedding timeouts.
- CORS regex: now matches root domain (worldmonitor.app) not just subdomains - Alert keywords: re-add important terms (military, drone strike, terror/cyber attack, evacuation order)
- Sparkle button (✨) in panel header triggers AI summary - Summary cached in localStorage for 10 minutes - Shows loading spinner while generating - Dismissable summary banner below header - Uses existing summarization fallback chain (Groq → OpenRouter → Browser T5)
Fixes 403 errors for: - Middle East: Al Arabiya, Arab News, Times of Israel, SCMP - Regional: Kyiv Independent, Moscow Times, News24 - Int'l Orgs: UN News, IAEA, WHO, CISA, Crisis Group - Other: Hacker News (news.ycombinator.com)
Railway's RSS proxy had only 5 domains in allowlist, causing 403 errors. Added all domains that use railwayRss() routing in feeds.ts: - Al Arabiya, Arab News, Times of Israel, SCMP - UN News, CISA, News24 - Plus IAEA, WHO, Crisis Group, Kyiv Independent, Moscow Times for future use
- Double MAX_RESULTS from 12 to 24 - Prioritize news over static infrastructure in search - Update News24 URL to post-redirect destination (feeds.capi24.com) - Add trailing slash to SCMP URL - Add feeds.capi24.com to both proxy allowlists
- Add keyword boosting for critical geopolitical terms (war, armada, military, iran, russia, ukraine, etc.) - +40 base + 10 per keyword - Shorten summaries: max 2 sentences, 40 words, max_tokens=80 - Critical terms now outrank generic "alert" stories
Add max-height, overflow-y auto, and flex-shrink to .panel-summary to prevent long summaries from pushing content out of view.
Base map tiles already display country names, so the custom Country Labels layer was duplicating functionality. Removed: - COUNTRY_LABELS array (~80 countries) from geo.ts - Toggle from layer controls in both Map.ts and DeckGLMap.ts - renderCountryLabels() and createCountryLabelsLayer() methods - 'countries' property from MapLayers type - All countries config entries from panels and variants - Related CSS styles
- Add headline deduplication (60% word similarity threshold) - Prevent prompt instructions from leaking into output - Increase max_tokens from 80 to 150 to prevent truncation - Add multi-tier keyword scoring: military (+80), flashpoint (+60), crisis (+30) - Demote business/tech news (0.3 penalty) when mixed with conflict keywords
- Cluster datacenters at zoom < 5, show IconLayer at higher zoom - Add renderDatacenterClusters() with HTML overlays - Add datacenterCluster popup type showing total chips/power stats - Add CSS styles for datacenter markers matching protest/techHQ pattern - Show cluster badge with count, expand to individual on click
- New /api/risk-scores endpoint computes and caches scores in Redis - Uses ACLED protest data + baseline geopolitical risk factors - 10-minute cache TTL shared across all users - CIIPanel and StrategicRiskPanel fetch cached scores on load - Learning Mode banner hidden when cached scores available - Eliminates 15-minute warmup for users
- Panel resize: drag bottom edge to span 1-4 grid rows, persisted to localStorage - Fix HTML5 drag conflict with resize using capture-phase listeners - Fix memory leaks in Panel.ts (document listeners now cleaned up) - Fix Railway ERR_HTTP_HEADERS_SENT crash with response flag pattern - Improve AI summarization prompts to focus on ONE dominant narrative - Add VIOLENCE_KEYWORDS and UNREST_KEYWORDS for better story discovery - Add combo bonus for flashpoint + unrest stories (e.g., Iran protests) - Relax 2-source filter for high-scoring critical stories (score > 100)
- New parallel-analysis.ts: runs browser ML alongside API summarization - Uses 6 perspectives: keywords, sentiment, entities (NER), novelty, velocity, sources - Detects disagreement between perspectives (flags potential missed stories) - Console dashboard logs analysis comparison for debugging - NER model (previously unused) now extracts entities for geopolitical scoring - Shows "ML DETECTED" section for stories keywords missed but ML flagged - Fix CII 15-min learning mode: bypasses when cached scores available
- Rewrite prompts to vary output (no more "The dominant narrative...") - Instruct model to lead with substance: location, action, impact - Add explicit rule: NEVER start with meta-commentary
Correlates news entities with map signals to identify "main characters" appearing across multiple intelligence streams: - Extract entities from all 80+ news sources via NER - Cross-reference with map signals (flights, vessels, outages, protests) - Score focal points where news + signals converge - Generate rich AI context for correlation-aware summarization New files: - focal-point-detector.ts: Core detection and scoring service - signal-aggregator.ts: Geographic signal aggregation UI shows top focal points with urgency badges and signal icons. AI prompts updated to leverage intelligence synthesis context.
- Add Market & Crypto Intelligence data layer section - Document macro signal analysis (7-signal BUY/CASH verdict, VWAP, Mayer Multiple) - Document stablecoin peg monitoring and BTC ETF flow estimation algorithms - Add strategic theater posture assessment (9 theaters) - Add infrastructure cascade BFS modeling with chokepoint dependencies - Add browser-side ML pipeline section (Transformers.js capabilities) - Add dual-deployment architecture diagram (Vercel + Railway) - Add caching architecture section (3-tier with stale-on-error) - Add security model section (8-layer defense table) - Update edge function count (30+ → 45+), tech stack, env vars - Add Cmd+K search, virtual scrolling, UCDP/HAPI to capabilities
- Extract close price arrays from Yahoo Finance chart API for indices/commodities - Switch CoinGecko crypto fetch to /coins/markets endpoint with 7d sparkline data - Render inline SVG sparklines color-coded green/red by price direction - Fix Vite dev proxy for CoinGecko (was hitting root instead of /api/v3/simple/price) - Add endpoint=markets support to CoinGecko edge function
- Implements layer caching to avoid rebuilding unchanged deck.gl layers - Adds cluster element DOM caching with position-based invalidation - Adds cluster result caching to prevent redundant spatial computations - Replaces setInterval with requestAnimationFrame for news pulse animation - Adds throttle, debounce, and rafSchedule utilities for render optimization - Removes hover tooltips in favor of click-based interactions (better performance) - Adds performance monitoring with console warnings for >16ms operations - Implements smart layer rebuilding only on zoom threshold crossings - Reduces cluster overlay opacity during map moves for smoother UX - Configures deck.gl with interleaved=false and useDevicePixels=false for performance Performance improvements: - Reduces redundant layer rebuilds by ~90% during map interactions - Eliminates tooltip hover callback overhead - Prevents cluster DOM thrashing with element reuse - 60fps target for map panning/zooming maintained
Adds a new "Gulf Economies" panel tracking all 6 GCC countries with live
data refreshed every 60 seconds via a dedicated edge function.
Panel features (3 tabs):
- Markets: Stock indices for Saudi TASI, Dubai DFM, Abu Dhabi ADX,
Qatar QSE, Kuwait KW15, Bahrain BAX, and Oman MSM30 — all with
sparklines and % change
- Currencies: SAR, AED, QAR, KWD, BHD, OMR vs USD exchange rates
- Oil: WTI and Brent crude prices (core Gulf economy driver)
New files:
- api/gulf-markets.js – Vercel edge function, fetches all GCC
symbols from Yahoo Finance in parallel
- src/services/gulf-markets.ts – TypeScript service with last-successful
cache for resilience
- src/components/GulfEconomyPanel.ts – Tabbed panel component
Also adds 10 curated Gulf economic news feeds (Arab News, GCC economy,
OPEC, Saudi Vision 2030, UAE markets, Qatar QIA, etc.) that flow into
the global news pool and Live News ticker.
https://claude.ai/code/session_014rCJ8bXKdgFCAFbVWn7MHt
|
@claude is attempting to deploy a commit to the Elie Team on Vercel. A member of the Team first needs to authorize it. |
|
Thank you @aa5064 will review later today after I finish improving caching all throughout the project, my $$$ are vanishing fast |
Of course! I can only imagine how much $$$ you are burning (: |
|
Thanks for the Gulf Economies concept! Great idea to track GCC markets — this is exactly the kind of regional financial coverage WorldMonitor needs. We can't merge this PR directly because the fork has diverged significantly (670 commits, 920 files changed), but we're going to implement this feature fresh on a new branch using the project's existing patterns (staggered Yahoo fetches, Upstash caching, circuit breakers, etc.). The new PR will reference this one for attribution. Thanks for the contribution! 🙏 |
… oil Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance: - 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM) - 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD) - 2 oil benchmarks (WTI, Brent) Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for rate-limit-safe sequential fetching. Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines, 8s delayed start, 60s polling. Registered in both full and finance variants. Inspired by #641 (credit: @aa5064, commit 684a73c).
… oil (#667) * feat(market): add Gulf Economies panel with GCC indices, currencies & oil Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance: - 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM) - 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD) - 2 oil benchmarks (WTI, Brent) Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for rate-limit-safe sequential fetching. Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines, 8s delayed start, 60s polling. Registered in both full and finance variants. Inspired by #641 (credit: @aa5064, commit 684a73c). * refactor: hoist MarketServiceClient to module scope, clarify memCache closure * fix(market): register list-gulf-quotes in RPC_CACHE_TIER map Explicitly set 'medium' cache tier for the new Gulf quotes endpoint, matching other market RPCs instead of relying on the implicit default. * refactor: extract shared miniSparkline to @/utils/sparkline Deduplicates the identical function from MarketPanel and GulfEconomiesPanel into a single shared utility. * fix(panels): disable gulf-economies by default on full variant Panel remains always-on for finance variant. Full variant users can manually enable it via panel settings.
… oil (#667) * feat(market): add Gulf Economies panel with GCC indices, currencies & oil Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance: - 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM) - 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD) - 2 oil benchmarks (WTI, Brent) Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for rate-limit-safe sequential fetching. Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines, 8s delayed start, 60s polling. Registered in both full and finance variants. Inspired by #641 (credit: @aa5064, commit 684a73c). * refactor: hoist MarketServiceClient to module scope, clarify memCache closure * fix(market): register list-gulf-quotes in RPC_CACHE_TIER map Explicitly set 'medium' cache tier for the new Gulf quotes endpoint, matching other market RPCs instead of relying on the implicit default. * refactor: extract shared miniSparkline to @/utils/sparkline Deduplicates the identical function from MarketPanel and GulfEconomiesPanel into a single shared utility. * fix(panels): disable gulf-economies by default on full variant Panel remains always-on for finance variant. Full variant users can manually enable it via panel settings.
… oil (koala73#667) * feat(market): add Gulf Economies panel with GCC indices, currencies & oil Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance: - 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM) - 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD) - 2 oil benchmarks (WTI, Brent) Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for rate-limit-safe sequential fetching. Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines, 8s delayed start, 60s polling. Registered in both full and finance variants. Inspired by koala73#641 (credit: @aa5064, commit 684a73c). * refactor: hoist MarketServiceClient to module scope, clarify memCache closure * fix(market): register list-gulf-quotes in RPC_CACHE_TIER map Explicitly set 'medium' cache tier for the new Gulf quotes endpoint, matching other market RPCs instead of relying on the implicit default. * refactor: extract shared miniSparkline to @/utils/sparkline Deduplicates the identical function from MarketPanel and GulfEconomiesPanel into a single shared utility. * fix(panels): disable gulf-economies by default on full variant Panel remains always-on for finance variant. Full variant users can manually enable it via panel settings.


Overview
This PR adds a dedicated Gulf Economies panel to WorldMonitor that tracks
the financial markets and economic indicators for all six Gulf Cooperation
Council (GCC) countries — Saudi Arabia, UAE, Qatar, Kuwait, Bahrain, and Oman
— with real-time data refreshed every 60 seconds.
Why This Matters
The Gulf region controls ~40% of the world's proven oil reserves and manages
over $4 trillion in sovereign wealth. The GCC markets are strategically
important but largely absent from existing Western-centric financial dashboards.
What Was Built
api/gulf-markets.js— New Vercel Edge FunctionFetches 15 Yahoo Finance symbols in parallel (8s timeout per symbol):
Returns price, 24h % change, and a 20-point sparkline per symbol.
CDN-cached for 60 seconds.
src/services/gulf-markets.ts— TypeScript ServiceThin fetch wrapper with a last-successful-result cache — if the API call
fails, the panel continues showing the last known data rather than going blank.
src/components/GulfEconomyPanel.ts— Tabbed Panel Component📈 Markets tab — All 7 GCC stock indices with:
💱 Currencies tab — Live SAR, AED, QAR, KWD, BHD, OMR vs USD rates
🛢️ Oil tab — WTI and Brent crude (primary GDP driver for all GCC economies)
src/config/feeds.ts— 10 Gulf Economic News FeedsNew
gulffeed category with:Feeds flow into the Live News ticker and AI Insights panel.
Data Sources
^TASI,^DFMGI,^FTFADGI,^QSI,^KW15,^BAX,^MSM)SARUSD=X,AEDUSD=X, etc.)CL=F,BZ=Ffutures)Files Changed