Skip to content

Batch market and RSS fetching with progressive updates#1

Merged
koala73 merged 1 commit into
mainfrom
codex/implement-parallel-api-fetching-for-stocks
Jan 10, 2026
Merged

Batch market and RSS fetching with progressive updates#1
koala73 merged 1 commit into
mainfrom
codex/implement-parallel-api-fetching-for-stocks

Conversation

@koala73

@koala73 koala73 commented Jan 9, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Stock and RSS fetching was fully sequential causing long initial load times and a poor user perception of performance.
  • The change aims to reduce perceived load by issuing requests in parallel batches while respecting rate limits.
  • Progressive rendering is required so the UI is updated as each batch completes instead of waiting for all requests.
  • The same batched pattern should be applied to RSS feeds to accelerate news population.

Description

  • Add a chunkArray helper in src/utils/index.ts to split lists into batches.
  • Update fetchMultipleStocks (src/services/markets.ts) to run batches via Promise.all, add options batchSize, delayMs, and onBatch callback, and keep a default delay of 2500ms to respect rate limits.
  • Update fetchCategoryFeeds (src/services/rss.ts) to fetch feeds in batches and expose an onBatch callback that provides progressively merged and sorted news items.
  • Wire the new onBatch callbacks in src/App.ts to render partial market and news results for stocks, sectors, commodities, and news categories as batches complete.

Testing

  • Started the dev server with npm run dev, which launched Vite successfully and served the app (succeeded).
  • Captured a browser screenshot using Playwright to verify progressive rendering (artifacts/worldmonitor-progressive.png) (succeeded).
  • Observed multiple network/proxy errors (ENETUNREACH) while fetching external APIs in the current environment, which prevented some live API responses (failed requests due to network environment).
  • No unit tests were added or executed as part of this change.

Codex Task

@vercel

vercel Bot commented Jan 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
worldmonitor Ready Ready Preview, Comment Jan 9, 2026 7:58pm

@koala73
koala73 merged commit cca7bc7 into main Jan 10, 2026
2 checks passed
@koala73
koala73 deleted the codex/implement-parallel-api-fetching-for-stocks branch January 12, 2026 18:47
koala73 added a commit that referenced this pull request Jan 18, 2026
… freshness

Assessment & Documentation:
- Add docs/GEOPOLITICAL_ASSESSMENT.md with full platform analysis
- Strategic improvement roadmap with prioritized recommendations

Quick Win #1 - Data Freshness (intelligence gaps):
- Add getIntelligenceGaps() and getIntelligenceGapSummary() to data-freshness.ts
- Add human-readable messages explaining what analysts CAN'T see
- Add hasCriticalGaps() for alert integration

Quick Win #2 - Escalation Scores:
- Add escalationScore (1-5), escalationTrend, escalationIndicators to Hotspot type
- Update 11 major hotspots with scores (Sahel, Haiti, Horn of Africa, Moscow,
  Beijing, Kyiv, Taipei, Tehran, Tel Aviv, Pyongyang, Sana'a)

Quick Win #3 - Signal Context ("Why It Matters"):
- Add SIGNAL_CONTEXT with whyItMatters, actionableInsight, confidenceNote
- Add getSignalContext() helper for all 10 signal types
- Explains analytical significance of each signal type

Quick Win #4 - Historical Context:
- Add HistoricalContext interface with lastMajorEvent, precedentCount,
  cyclicalRisk fields
- Add whyItMatters field to Hotspot type
- Update major hotspots with historical precedents and geopolitical significance

Quick Win #5 - Propaganda Risk Flags:
- Add PropagandaRisk type and SourceRiskProfile interface
- Add SOURCE_PROPAGANDA_RISK mapping for state media (Xinhua, TASS, RT, CGTN)
- Add getSourcePropagandaRisk() and isStateAffiliatedSource() helpers
- Flag medium-risk state-affiliated sources (Al Jazeera, France 24, DW, etc.)
@SebastienMelki SebastienMelki added performance Performance optimization area: markets Markets, commodities, crypto, finance area: feeds RSS feeds, news aggregation labels Feb 17, 2026
facusturla pushed a commit to facusturla/worldmonitor that referenced this pull request Feb 27, 2026
…i-fetching-for-stocks

Batch market and RSS fetching with progressive updates
facusturla pushed a commit to facusturla/worldmonitor that referenced this pull request Feb 27, 2026
… freshness

Assessment & Documentation:
- Add docs/GEOPOLITICAL_ASSESSMENT.md with full platform analysis
- Strategic improvement roadmap with prioritized recommendations

Quick Win koala73#1 - Data Freshness (intelligence gaps):
- Add getIntelligenceGaps() and getIntelligenceGapSummary() to data-freshness.ts
- Add human-readable messages explaining what analysts CAN'T see
- Add hasCriticalGaps() for alert integration

Quick Win koala73#2 - Escalation Scores:
- Add escalationScore (1-5), escalationTrend, escalationIndicators to Hotspot type
- Update 11 major hotspots with scores (Sahel, Haiti, Horn of Africa, Moscow,
  Beijing, Kyiv, Taipei, Tehran, Tel Aviv, Pyongyang, Sana'a)

Quick Win koala73#3 - Signal Context ("Why It Matters"):
- Add SIGNAL_CONTEXT with whyItMatters, actionableInsight, confidenceNote
- Add getSignalContext() helper for all 10 signal types
- Explains analytical significance of each signal type

Quick Win koala73#4 - Historical Context:
- Add HistoricalContext interface with lastMajorEvent, precedentCount,
  cyclicalRisk fields
- Add whyItMatters field to Hotspot type
- Update major hotspots with historical precedents and geopolitical significance

Quick Win koala73#5 - Propaganda Risk Flags:
- Add PropagandaRisk type and SourceRiskProfile interface
- Add SOURCE_PROPAGANDA_RISK mapping for state media (Xinhua, TASS, RT, CGTN)
- Add getSourcePropagandaRisk() and isStateAffiliatedSource() helpers
- Flag medium-risk state-affiliated sources (Al Jazeera, France 24, DW, etc.)
koala73 added a commit that referenced this pull request Apr 11, 2026
Addresses 2 P1 review findings on PR #2942.

P1 #1: Region pills keyed to labels that do not match persisted Forecast.region.
  The seed-forecasts writes Forecast.region with country names (Israel, Iran,
  Taiwan, United States...) and theater/market values (Red Sea, Black Sea,
  Baltic Sea, South China Sea, Eastern Mediterranean, Strait of Hormuz,
  Western Pacific...). The server RPC only does a substring .includes() match,
  so pills like 'Middle East' and 'Europe' miss most of the data. The
  seed already computes macroRegion via MACRO_REGION_MAP but that field is
  not in the Forecast proto, so the client never sees it.

  Fix: mirror MACRO_REGION_MAP to new shared/forecast-macro-regions.js and
  filter client-side by macro region. Pills now use MENA, EAST_ASIA, EUROPE,
  AMERICAS, SOUTH_ASIA, AFRICA as the six macro regions that match the
  publisher's taxonomy.

P1 #2: Region filter state lost on refresh.
  selectedRegion was only consumed by refetchForRegion(). When the data
  loader or scheduler called updateForecasts(fullForecasts), this.forecasts
  was overwritten and the region filter disappeared. render() did not know
  about the filter.

  Fix: apply region filter inside the render pipeline (matching the existing
  activeDomain pattern). this.forecasts stays as the full unfiltered set.
  Clicking a pill no longer issues an RPC; it just updates selectedRegion
  and re-renders. Removed the regionFetchSeq counter since no network call
  fires.

  Side effect: improved perf (no redundant RPC on pill clicks) and instant
  response. The getForecasts RPC is called once on initial load, same as
  before.
koala73 added a commit that referenced this pull request Apr 11, 2026
P2 #1 — Mexico bbox mismatch between airport and flight classifiers

airportToSnapshotRegion() routes Mexico to north-america by country,
but latLonToSnapshotRegion()'s NA bbox started at lat 20°N. Mexico City
(19.43°N), Guadalajara (20.67°N), and most Mexican airspace fell into
latam, so NA's reroute_intensity understated its actual military
pressure and the two classifiers disagreed on the same country.

Fix: lower the NA bbox southern edge to lat 16.0°N. That captures
every major Mexican city and state capital (Tuxtla Gutiérrez at
16.75°N is the southernmost) while still routing Guatemala City
(14.6°N), San Salvador, Belize, and Honduras to latam. Tiny airports
at lat ~14.9°N (Tapachula in southern Chiapas) route to latam —
acceptable because they're not in the monitored set and don't carry
meaningful military traffic.

Added 8 regression tests asserting classifier parity across major
Mexican cities and correct routing of Central American capitals.

P2 #2 — seed-meta freshness path for undated payloads

classifyInputs() was treating undated payloads as "fresh" via its
fallback branch, so FAA/NOTAM/AviationStack/GPS-jam stalls would
never drag down snapshot_confidence. None of those four payloads
carry a top-level fetchedAt field; the seeders track freshness in
companion seed-meta:* keys only.

Fix: added an optional `metaKey` field to FreshnessRegistry entries.
classifyInputs() now prefers the metaKey companion's fetchedAt when
the spec declares one, falling back to the primary payload's
top-level timestamp (existing behavior) and finally to "fresh"
(existing fallback for legacy undated payloads).

Wired through the full pipeline:
  - freshness.mjs: added metaKey type, ALL_META_KEYS export, and
    metaPayloads parameter to classifyInputs()
  - snapshot-meta.mjs: buildPreMeta() forwards metaSources
  - seed-regional-snapshots.mjs: readAllInputs() now fetches both
    ALL_INPUT_KEYS and ALL_META_KEYS in a single pipeline and
    returns { sources, metaSources }; main() destructures and
    passes metaSources through computeSnapshot → buildPreMeta
  - computeSnapshot() signature gained an optional metaSources arg
    (defaults to {} for back-compat with existing tests)

Registered metaKey for each mobility input:
  aviation:delays:faa:v1       → seed-meta:aviation:faa
  aviation:delays:intl:v3      → seed-meta:aviation:intl
  aviation:notam:closures:v2   → seed-meta:aviation:notam
  intelligence:gpsjam:v2       → seed-meta:intelligence:gpsjam

military:flights:v1 does NOT declare a metaKey because its payload
already carries top-level fetchedAt — this is asserted in a test.

Added 11 regression tests covering:
  - every undated mobility input has a metaKey
  - military:flights:v1 does NOT need a metaKey
  - ALL_META_KEYS aggregation
  - undated payload + fresh meta → fresh
  - undated payload + STALE meta → stale (the core bug fix)
  - undated payload + missing meta → falls back to fresh
  - missing payload → missing regardless of meta
  - legacy payloads with top-level timestamp still work
  - meta without fetchedAt falls through to payload timestamp

## Verification

- node --test tests/regional-snapshot-mobility.test.mjs: 54/54 pass
  (35 original + 19 new regression tests)
- npm run test:data: 4556/4556 pass
- npm run typecheck: clean
- biome lint on touched files: clean
This was referenced Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: feeds RSS feeds, news aggregation area: markets Markets, commodities, crypto, finance codex performance Performance optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants