Skip to content

feat: Gulf Stock Market & Economy Tracker for all 6 GCC Countries#641

Closed
aa5064 wants to merge 670 commits into
koala73:mainfrom
aa5064:claude/world-monitor-commit-9zGS7
Closed

feat: Gulf Stock Market & Economy Tracker for all 6 GCC Countries#641
aa5064 wants to merge 670 commits into
koala73:mainfrom
aa5064:claude/world-monitor-commit-9zGS7

Conversation

@aa5064

@aa5064 aa5064 commented Mar 1, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR adds a dedicated Gulf Economies panel to WorldMonitor that tracks
the financial markets and economic indicators for all six Gulf Cooperation
Council (GCC) countries — Saudi Arabia, UAE, Qatar, Kuwait, Bahrain, and Oman
— with real-time data refreshed every 60 seconds.

Why This Matters

The Gulf region controls ~40% of the world's proven oil reserves and manages
over $4 trillion in sovereign wealth. The GCC markets are strategically
important but largely absent from existing Western-centric financial dashboards.

Country Exchange Index Notes
🇸🇦 Saudi Arabia Tadawul TASI Largest Arab stock market (~$3T market cap)
🇦🇪 UAE (Dubai) Dubai Financial Market DFMGI Major re-export and logistics hub
🇦🇪 UAE (Abu Dhabi) Abu Dhabi Securities Exchange ADX Home to ADNOC, UAE's oil giant
🇶🇦 Qatar Qatar Stock Exchange QSE LNG-driven economy, world's highest GDP/capita
🇰🇼 Kuwait Boursa Kuwait KW15 Among the oldest Gulf exchanges (est. 1977)
🇧🇭 Bahrain Bahrain Bourse BAX Regional financial services hub
🇴🇲 Oman Muscat Stock Exchange MSM30 Diversification-focused via Vision 2040

What Was Built

api/gulf-markets.js — New Vercel Edge Function

Fetches 15 Yahoo Finance symbols in parallel (8s timeout per symbol):

  • 7 stock market indices (one per GCC country)
  • 6 currency pairs (each GCC currency vs USD)
  • 2 oil benchmarks (WTI + Brent crude)

Returns price, 24h % change, and a 20-point sparkline per symbol.
CDN-cached for 60 seconds.

src/services/gulf-markets.ts — TypeScript Service

Thin fetch wrapper with a last-successful-result cache — if the API call
fails, the panel continues showing the last known data rather than going blank.

src/components/GulfEconomyPanel.ts — Tabbed Panel Component

📈 Markets tab — All 7 GCC stock indices with:

  • Country flag + exchange name
  • Current index level (locale-formatted)
  • 24h % change with ▲/▼ arrows and red/green coloring
  • 20-point mini sparkline (past month trend)

💱 Currencies tab — Live SAR, AED, QAR, KWD, BHD, OMR vs USD rates

🛢️ Oil tab — WTI and Brent crude (primary GDP driver for all GCC economies)

src/config/feeds.ts — 10 Gulf Economic News Feeds

New gulf feed category with:

  • Arab News (Saudi Arabia's leading English business daily)
  • Saudi Vision 2030 / Aramco coverage
  • UAE market news (DFM, ADX, Abu Dhabi investment)
  • Qatar QIA sovereign wealth fund updates
  • Kuwait Investment Authority news
  • GCC-wide economy roundups + OPEC production news
  • Gulf mega-project updates (NEOM, ADNOC, etc.)

Feeds flow into the Live News ticker and AI Insights panel.

Data Sources

Data Source Refresh
Stock indices Yahoo Finance (^TASI, ^DFMGI, ^FTFADGI, ^QSI, ^KW15, ^BAX, ^MSM) 60s
Currency rates Yahoo Finance (SARUSD=X, AEDUSD=X, etc.) 60s
Oil prices Yahoo Finance (CL=F, BZ=F futures) 60s
News Google News RSS + Arab News RSS 5 min

Files Changed

- Re-enable the ML worker import that was disabled
- Remove @xenova/transformers from rollup externals so it gets bundled
- Previous config had MLWorkerClass set to null, causing "not a constructor" error
- Show "Initializing ML models..." when worker not ready
- Show "Analyzing sentiment..." during sentiment classification
- Show "Generating world brief..." during summarization
- Add animated spinner and pulsing text for visual feedback
- Add /api/groq-summarize.js - Llama 3.3 70B (1000 req/day free)
- Add /api/openrouter-summarize.js - Fallback (50 req/day free)
- Create summarization service with fallback: Groq -> OpenRouter -> Browser T5
- Browser T5 only loads when cloud APIs fail (lazy loading)
- Add detailed progress bar with Step X/Y indicator
- Show provider badge (groq/openrouter) on World Brief
- Cache summaries for 30 min to respect rate limits
- Cooldown increased to 2 min between brief generations

Required env vars for Vercel:
- GROQ_API_KEY (get from console.groq.com)
- OPENROUTER_API_KEY (get from openrouter.ai)
- Add Upstash Redis caching to groq-summarize.js and openrouter-summarize.js
- Switch to llama-3.1-8b-instant (14.4K/day vs 1K for 70b)
- Cross-user cache deduplication with 24h TTL
- Remove client-side cache (server handles all caching now)

Improve AI Insights story selection:
- Composite importance score: sources × velocity × recency + alert bonus
- Source diversity cap: max 3 stories from same source
- Recency decay: newer stories rank higher (12h half-life)

Scalability: ~144x headroom (was hitting 1K/day limit)
…lusions

- Remove 'breaking', 'emergency' from alert keywords (too generic)
- Add ALERT_EXCLUSIONS for lifestyle/entertainment content
- Remove confusing 'Major' tier label (was showing for T2 sources)
- Tier badge now just shows dot for T2, 'Wire' for T1
World Brief in AI Insights already provides panel-level summarization.
Also reduced ML batch size and timeout to prevent embedding timeouts.
- CORS regex: now matches root domain (worldmonitor.app) not just subdomains
- Alert keywords: re-add important terms (military, drone strike, terror/cyber attack, evacuation order)
- Sparkle button (✨) in panel header triggers AI summary
- Summary cached in localStorage for 10 minutes
- Shows loading spinner while generating
- Dismissable summary banner below header
- Uses existing summarization fallback chain (Groq → OpenRouter → Browser T5)
Fixes 403 errors for:
- Middle East: Al Arabiya, Arab News, Times of Israel, SCMP
- Regional: Kyiv Independent, Moscow Times, News24
- Int'l Orgs: UN News, IAEA, WHO, CISA, Crisis Group
- Other: Hacker News (news.ycombinator.com)
Railway's RSS proxy had only 5 domains in allowlist, causing 403 errors.
Added all domains that use railwayRss() routing in feeds.ts:
- Al Arabiya, Arab News, Times of Israel, SCMP
- UN News, CISA, News24
- Plus IAEA, WHO, Crisis Group, Kyiv Independent, Moscow Times for future use
- Double MAX_RESULTS from 12 to 24
- Prioritize news over static infrastructure in search
- Update News24 URL to post-redirect destination (feeds.capi24.com)
- Add trailing slash to SCMP URL
- Add feeds.capi24.com to both proxy allowlists
- Add keyword boosting for critical geopolitical terms (war, armada,
  military, iran, russia, ukraine, etc.) - +40 base + 10 per keyword
- Shorten summaries: max 2 sentences, 40 words, max_tokens=80
- Critical terms now outrank generic "alert" stories
Add max-height, overflow-y auto, and flex-shrink to .panel-summary
to prevent long summaries from pushing content out of view.
Base map tiles already display country names, so the custom
Country Labels layer was duplicating functionality.

Removed:
- COUNTRY_LABELS array (~80 countries) from geo.ts
- Toggle from layer controls in both Map.ts and DeckGLMap.ts
- renderCountryLabels() and createCountryLabelsLayer() methods
- 'countries' property from MapLayers type
- All countries config entries from panels and variants
- Related CSS styles
- Add headline deduplication (60% word similarity threshold)
- Prevent prompt instructions from leaking into output
- Increase max_tokens from 80 to 150 to prevent truncation
- Add multi-tier keyword scoring: military (+80), flashpoint (+60), crisis (+30)
- Demote business/tech news (0.3 penalty) when mixed with conflict keywords
- Cluster datacenters at zoom < 5, show IconLayer at higher zoom
- Add renderDatacenterClusters() with HTML overlays
- Add datacenterCluster popup type showing total chips/power stats
- Add CSS styles for datacenter markers matching protest/techHQ pattern
- Show cluster badge with count, expand to individual on click
- New /api/risk-scores endpoint computes and caches scores in Redis
- Uses ACLED protest data + baseline geopolitical risk factors
- 10-minute cache TTL shared across all users
- CIIPanel and StrategicRiskPanel fetch cached scores on load
- Learning Mode banner hidden when cached scores available
- Eliminates 15-minute warmup for users
- Panel resize: drag bottom edge to span 1-4 grid rows, persisted to localStorage
- Fix HTML5 drag conflict with resize using capture-phase listeners
- Fix memory leaks in Panel.ts (document listeners now cleaned up)
- Fix Railway ERR_HTTP_HEADERS_SENT crash with response flag pattern
- Improve AI summarization prompts to focus on ONE dominant narrative
- Add VIOLENCE_KEYWORDS and UNREST_KEYWORDS for better story discovery
- Add combo bonus for flashpoint + unrest stories (e.g., Iran protests)
- Relax 2-source filter for high-scoring critical stories (score > 100)
- New parallel-analysis.ts: runs browser ML alongside API summarization
- Uses 6 perspectives: keywords, sentiment, entities (NER), novelty, velocity, sources
- Detects disagreement between perspectives (flags potential missed stories)
- Console dashboard logs analysis comparison for debugging
- NER model (previously unused) now extracts entities for geopolitical scoring
- Shows "ML DETECTED" section for stories keywords missed but ML flagged
- Fix CII 15-min learning mode: bypasses when cached scores available
- Rewrite prompts to vary output (no more "The dominant narrative...")
- Instruct model to lead with substance: location, action, impact
- Add explicit rule: NEVER start with meta-commentary
Correlates news entities with map signals to identify "main characters"
appearing across multiple intelligence streams:

- Extract entities from all 80+ news sources via NER
- Cross-reference with map signals (flights, vessels, outages, protests)
- Score focal points where news + signals converge
- Generate rich AI context for correlation-aware summarization

New files:
- focal-point-detector.ts: Core detection and scoring service
- signal-aggregator.ts: Geographic signal aggregation

UI shows top focal points with urgency badges and signal icons.
AI prompts updated to leverage intelligence synthesis context.
koala73 and others added 17 commits February 11, 2026 19:06
- Add Market & Crypto Intelligence data layer section
- Document macro signal analysis (7-signal BUY/CASH verdict, VWAP, Mayer Multiple)
- Document stablecoin peg monitoring and BTC ETF flow estimation algorithms
- Add strategic theater posture assessment (9 theaters)
- Add infrastructure cascade BFS modeling with chokepoint dependencies
- Add browser-side ML pipeline section (Transformers.js capabilities)
- Add dual-deployment architecture diagram (Vercel + Railway)
- Add caching architecture section (3-tier with stale-on-error)
- Add security model section (8-layer defense table)
- Update edge function count (30+ → 45+), tech stack, env vars
- Add Cmd+K search, virtual scrolling, UCDP/HAPI to capabilities
- Extract close price arrays from Yahoo Finance chart API for indices/commodities
- Switch CoinGecko crypto fetch to /coins/markets endpoint with 7d sparkline data
- Render inline SVG sparklines color-coded green/red by price direction
- Fix Vite dev proxy for CoinGecko (was hitting root instead of /api/v3/simple/price)
- Add endpoint=markets support to CoinGecko edge function
- Implements layer caching to avoid rebuilding unchanged deck.gl layers
- Adds cluster element DOM caching with position-based invalidation
- Adds cluster result caching to prevent redundant spatial computations
- Replaces setInterval with requestAnimationFrame for news pulse animation
- Adds throttle, debounce, and rafSchedule utilities for render optimization
- Removes hover tooltips in favor of click-based interactions (better performance)
- Adds performance monitoring with console warnings for >16ms operations
- Implements smart layer rebuilding only on zoom threshold crossings
- Reduces cluster overlay opacity during map moves for smoother UX
- Configures deck.gl with interleaved=false and useDevicePixels=false for performance

Performance improvements:
- Reduces redundant layer rebuilds by ~90% during map interactions
- Eliminates tooltip hover callback overhead
- Prevents cluster DOM thrashing with element reuse
- 60fps target for map panning/zooming maintained
Adds a new "Gulf Economies" panel tracking all 6 GCC countries with live
data refreshed every 60 seconds via a dedicated edge function.

Panel features (3 tabs):
- Markets: Stock indices for Saudi TASI, Dubai DFM, Abu Dhabi ADX,
  Qatar QSE, Kuwait KW15, Bahrain BAX, and Oman MSM30 — all with
  sparklines and % change
- Currencies: SAR, AED, QAR, KWD, BHD, OMR vs USD exchange rates
- Oil: WTI and Brent crude prices (core Gulf economy driver)

New files:
- api/gulf-markets.js        – Vercel edge function, fetches all GCC
                               symbols from Yahoo Finance in parallel
- src/services/gulf-markets.ts – TypeScript service with last-successful
                               cache for resilience
- src/components/GulfEconomyPanel.ts – Tabbed panel component

Also adds 10 curated Gulf economic news feeds (Arab News, GCC economy,
OPEC, Saudi Vision 2030, UAE markets, Qatar QIA, etc.) that flow into
the global news pool and Live News ticker.

https://claude.ai/code/session_014rCJ8bXKdgFCAFbVWn7MHt
@vercel

vercel Bot commented Mar 1, 2026

Copy link
Copy Markdown

@claude is attempting to deploy a commit to the Elie Team on Vercel.

A member of the Team first needs to authorize it.

@aa5064 aa5064 changed the title Major expansion: Intelligence correlation, tech variant, and 50+ new data sources feat: Gulf Stock Market & Economy Tracker for all 6 GCC Countries Mar 1, 2026
@koala73

koala73 commented Mar 1, 2026

Copy link
Copy Markdown
Owner

Thank you @aa5064 will review later today after I finish improving caching all throughout the project, my $$$ are vanishing fast

@aa5064

aa5064 commented Mar 1, 2026

Copy link
Copy Markdown
Contributor Author

Thank you @aa5064 will review later today after I finish improving caching all throughout the project, my $$$ are vanishing fast

Of course! I can only imagine how much $$$ you are burning (:

@koala73

koala73 commented Mar 1, 2026

Copy link
Copy Markdown
Owner

Very cool this going in the finance variant, did you see
CleanShot 2026-03-01 at 17 29 50@2x

@koala73

koala73 commented Mar 1, 2026

Copy link
Copy Markdown
Owner

Thanks for the Gulf Economies concept! Great idea to track GCC markets — this is exactly the kind of regional financial coverage WorldMonitor needs.

We can't merge this PR directly because the fork has diverged significantly (670 commits, 920 files changed), but we're going to implement this feature fresh on a new branch using the project's existing patterns (staggered Yahoo fetches, Upstash caching, circuit breakers, etc.).

The new PR will reference this one for attribution. Thanks for the contribution! 🙏

@koala73 koala73 closed this Mar 1, 2026
koala73 added a commit that referenced this pull request Mar 1, 2026
… oil

Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance:
- 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM)
- 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD)
- 2 oil benchmarks (WTI, Brent)

Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis
via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for
rate-limit-safe sequential fetching.

Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines,
8s delayed start, 60s polling. Registered in both full and finance
variants.

Inspired by #641 (credit: @aa5064, commit 684a73c).
koala73 added a commit that referenced this pull request Mar 1, 2026
… oil (#667)

* feat(market): add Gulf Economies panel with GCC indices, currencies & oil

Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance:
- 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM)
- 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD)
- 2 oil benchmarks (WTI, Brent)

Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis
via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for
rate-limit-safe sequential fetching.

Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines,
8s delayed start, 60s polling. Registered in both full and finance
variants.

Inspired by #641 (credit: @aa5064, commit 684a73c).

* refactor: hoist MarketServiceClient to module scope, clarify memCache closure

* fix(market): register list-gulf-quotes in RPC_CACHE_TIER map

Explicitly set 'medium' cache tier for the new Gulf quotes endpoint,
matching other market RPCs instead of relying on the implicit default.

* refactor: extract shared miniSparkline to @/utils/sparkline

Deduplicates the identical function from MarketPanel and GulfEconomiesPanel
into a single shared utility.

* fix(panels): disable gulf-economies by default on full variant

Panel remains always-on for finance variant. Full variant users can
manually enable it via panel settings.
koala73 added a commit that referenced this pull request Mar 1, 2026
… oil (#667)

* feat(market): add Gulf Economies panel with GCC indices, currencies & oil

Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance:
- 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM)
- 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD)
- 2 oil benchmarks (WTI, Brent)

Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis
via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for
rate-limit-safe sequential fetching.

Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines,
8s delayed start, 60s polling. Registered in both full and finance
variants.

Inspired by #641 (credit: @aa5064, commit 684a73c).

* refactor: hoist MarketServiceClient to module scope, clarify memCache closure

* fix(market): register list-gulf-quotes in RPC_CACHE_TIER map

Explicitly set 'medium' cache tier for the new Gulf quotes endpoint,
matching other market RPCs instead of relying on the implicit default.

* refactor: extract shared miniSparkline to @/utils/sparkline

Deduplicates the identical function from MarketPanel and GulfEconomiesPanel
into a single shared utility.

* fix(panels): disable gulf-economies by default on full variant

Panel remains always-on for finance variant. Full variant users can
manually enable it via panel settings.
@aa5064

aa5064 commented Mar 1, 2026

Copy link
Copy Markdown
Contributor Author

Very cool this going in the finance variant, did you see CleanShot 2026-03-01 at 17 29 50@2x

Yes! its cool. Also, I am trying to find a way to make the GCC investment variant live.

matthewvecchione1-ops pushed a commit to matthewvecchione1-ops/worldmonitor that referenced this pull request Mar 4, 2026
… oil (koala73#667)

* feat(market): add Gulf Economies panel with GCC indices, currencies & oil

Add a new panel tracking 15 Gulf/GCC financial symbols via Yahoo Finance:
- 7 stock indices (TASI, DFMGI, ADX, QSI, Bahrain, Kuwait, MSM)
- 6 currencies (SAR, AED, QAR, KWD, BHD, OMR vs USD)
- 2 oil benchmarks (WTI, Brent)

Server: new ListGulfQuotes RPC with 3-tier caching (in-memory + Redis
via cachedFetchJson + stale fallback). Uses fetchYahooQuotesBatch for
rate-limit-safe sequential fetching.

Frontend: GulfEconomiesPanel with 3 collapsible sections, sparklines,
8s delayed start, 60s polling. Registered in both full and finance
variants.

Inspired by koala73#641 (credit: @aa5064, commit 684a73c).

* refactor: hoist MarketServiceClient to module scope, clarify memCache closure

* fix(market): register list-gulf-quotes in RPC_CACHE_TIER map

Explicitly set 'medium' cache tier for the new Gulf quotes endpoint,
matching other market RPCs instead of relying on the implicit default.

* refactor: extract shared miniSparkline to @/utils/sparkline

Deduplicates the identical function from MarketPanel and GulfEconomiesPanel
into a single shared utility.

* fix(panels): disable gulf-economies by default on full variant

Panel remains always-on for finance variant. Full variant users can
manually enable it via panel settings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants