Skip to content

feat: add RAG-powered analysis with local vector store#647

Closed
Yigtwxx wants to merge 1 commit into
koala73:mainfrom
Yigtwxx:feat/rag-powered-analysis
Closed

feat: add RAG-powered analysis with local vector store#647
Yigtwxx wants to merge 1 commit into
koala73:mainfrom
Yigtwxx:feat/rag-powered-analysis

Conversation

@Yigtwxx

@Yigtwxx Yigtwxx commented Mar 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Added a client-side RAG (Retrieval-Augmented Generation) pipeline that gives country intelligence briefs historical context from previously seen headlines. Here's the flow:

  • When RSS feeds are parsed, headlines are embedded in the background via the ML worker and stored as vectors in IndexedDB
  • When a country brief is requested, current headlines get embedded and searched against the local vector store for similar past events
  • The top-3 matching historical events are appended to the context snapshot sent to the Groq-powered intel brief endpoint

The new vector-store.ts module handles all the IndexedDB operations — cosine similarity search, TTL-based pruning, and storage quota error handling.

On the client side, country-intel.ts now enriches the existing buildBriefContextSnapshot output with RAG-retrieved historical events before passing it to fetchCountryIntelBrief. The RAG context flows through the existing context query param, so no proto or server changes were needed.

Also caught and fixed two issues during review:

  • The e2e test was deleting the wrong IndexedDB name in beforeEach (wm-vector-store instead of worldmonitor_vector_store)
  • rss.ts was importing hashString from server/_shared/hash, which was the only cross-boundary import in the entire src/ tree — moved it to src/utils/hash.ts

Type of change

  • New feature

Affected areas

  • News panels / RSS feeds
  • AI Insights / World Brief

Checklist

  • Tested on worldmonitor.app variant
  • Tested on tech.worldmonitor.app variant (if applicable)
  • New RSS feed domains added to api/rss-proxy.js allowlist (if adding feeds)
  • No API keys or secrets committed
  • TypeScript compiles without errors (npm run typecheck)

Screenshots

No UI changes — this is a data pipeline addition.

@vercel

vercel Bot commented Mar 1, 2026

Copy link
Copy Markdown

@Yigtwxx is attempting to deploy a commit to the Elie Team on Vercel.

A member of the Team first needs to authorize it.

- Add IndexedDB-backed vector store (src/services/vector-store.ts) with
  cosine similarity search, TTL-based pruning, and quota error handling
- Vectorize RSS headlines in background via ML worker embeddings
- Enrich country intel briefs with historical context from local vector store
- Add optional geoContext field to GetCountryIntelBriefRequest proto
- Update generated client/server stubs for new proto field
- Add e2e test for vector store CRUD and similarity search
- Add e2e/tsconfig.json extending root config
- Fix cross-boundary import: move hashString to src/utils/hash.ts
- Fix e2e test: correct IndexedDB name in beforeEach cleanup
@koala73

koala73 commented Mar 1, 2026

Copy link
Copy Markdown
Owner

Hey @Yigtwxx — thanks for the original contribution! I've taken your concept and redesigned the implementation to run entirely in the Web Worker (zero main-thread memory for vectors). The new PR is #675. Your work is credited there. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants