feat(news): server-side feed aggregation to cut Vercel costs ~70-95%#622
Merged
Conversation
…y ~95%
Phase 1: Force CDN caching on rss-proxy (s-maxage=300 for 2xx, short
TTL for errors) — fixes bug where upstream no-cache headers were passed
through verbatim, defeating Vercel CDN.
Phase 2: Add ListFeedDigest RPC that aggregates all feeds server-side
into a single Redis-cached response. Client makes 1 request instead of
~90 per cycle. Includes circuit breaker with persistent cache fallback,
per-feed AI reclassification, and headline ingestion parity.
Phase 3: Increase polling interval from 5min to 7min to offset CDN
cache alignment.
New files:
- proto/worldmonitor/news/v1/list_feed_digest.proto
- server/worldmonitor/news/v1/{_feeds,_classifier,list-feed-digest}.ts
- src/services/ai-classify-queue.ts (extracted from rss.ts)
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Take main's 'static' tier for list-airport-delays.
koala73
added a commit
that referenced
this pull request
Mar 1, 2026
PR #622 (server-side feed aggregation) only ported 8 of 14 finance categories to VARIANT_FEEDS. The 6 omitted categories (derivatives, fintech, regulation, institutional, analysis, gccNews) caused "UNAVAILABLE" panels on finance.worldmonitor.app. Restores all feeds to match the pre-migration frontend config exactly. Also adds missing public/favico/finance/ directory (404s on favicon).
3 tasks
koala73
added a commit
that referenced
this pull request
Mar 1, 2026
PR #622 (server-side feed aggregation) only ported 8 of 14 finance categories to VARIANT_FEEDS. The 6 omitted categories (derivatives, fintech, regulation, institutional, analysis, gccNews) caused "UNAVAILABLE" panels on finance.worldmonitor.app. Restores all feeds to match the pre-migration frontend config exactly. Also adds missing public/favico/finance/ directory (404s on favicon).
koala73
added a commit
that referenced
this pull request
Mar 1, 2026
#654) PR #622 (server-side feed aggregation) only ported 8 of 14 finance categories to VARIANT_FEEDS. The 6 omitted categories (derivatives, fintech, regulation, institutional, analysis, gccNews) caused "UNAVAILABLE" panels on finance.worldmonitor.app. Restores all feeds to match the pre-migration frontend config exactly. Also adds missing public/favico/finance/ directory (404s on favicon).
koala73
added a commit
that referenced
this pull request
Mar 1, 2026
…y ~95% (#622) Phase 1: Force CDN caching on rss-proxy (s-maxage=300 for 2xx, short TTL for errors) — fixes bug where upstream no-cache headers were passed through verbatim, defeating Vercel CDN. Phase 2: Add ListFeedDigest RPC that aggregates all feeds server-side into a single Redis-cached response. Client makes 1 request instead of ~90 per cycle. Includes circuit breaker with persistent cache fallback, per-feed AI reclassification, and headline ingestion parity. Phase 3: Increase polling interval from 5min to 7min to offset CDN cache alignment. New files: - proto/worldmonitor/news/v1/list_feed_digest.proto - server/worldmonitor/news/v1/{_feeds,_classifier,list-feed-digest}.ts - src/services/ai-classify-queue.ts (extracted from rss.ts)
koala73
added a commit
that referenced
this pull request
Mar 1, 2026
#654) PR #622 (server-side feed aggregation) only ported 8 of 14 finance categories to VARIANT_FEEDS. The 6 omitted categories (derivatives, fintech, regulation, institutional, analysis, gccNews) caused "UNAVAILABLE" panels on finance.worldmonitor.app. Restores all feeds to match the pre-migration frontend config exactly. Also adds missing public/favico/finance/ directory (404s on favicon).
matthewvecchione1-ops
pushed a commit
to matthewvecchione1-ops/worldmonitor
that referenced
this pull request
Mar 4, 2026
…y ~95% (koala73#622) Phase 1: Force CDN caching on rss-proxy (s-maxage=300 for 2xx, short TTL for errors) — fixes bug where upstream no-cache headers were passed through verbatim, defeating Vercel CDN. Phase 2: Add ListFeedDigest RPC that aggregates all feeds server-side into a single Redis-cached response. Client makes 1 request instead of ~90 per cycle. Includes circuit breaker with persistent cache fallback, per-feed AI reclassification, and headline ingestion parity. Phase 3: Increase polling interval from 5min to 7min to offset CDN cache alignment. New files: - proto/worldmonitor/news/v1/list_feed_digest.proto - server/worldmonitor/news/v1/{_feeds,_classifier,list-feed-digest}.ts - src/services/ai-classify-queue.ts (extracted from rss.ts)
matthewvecchione1-ops
pushed a commit
to matthewvecchione1-ops/worldmonitor
that referenced
this pull request
Mar 4, 2026
koala73#654) PR koala73#622 (server-side feed aggregation) only ported 8 of 14 finance categories to VARIANT_FEEDS. The 6 omitted categories (derivatives, fintech, regulation, institutional, analysis, gccNews) caused "UNAVAILABLE" panels on finance.worldmonitor.app. Restores all feeds to match the pre-migration frontend config exactly. Also adds missing public/favico/finance/ directory (404s on favicon).
Yachtman25
added a commit
to Dabblecraft2DevTeam/worldmonitor
that referenced
this pull request
Mar 12, 2026
Yachtman25
added a commit
to Dabblecraft2DevTeam/worldmonitor
that referenced
this pull request
Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the root cause of ~$5,740/10-day Vercel bill by eliminating redundant RSS feed fetching:
api/rss-proxy.jswas passing upstreamCache-Controlheaders (oftenno-cache/private) through to Vercel CDN, defeating edge caching. Now forcess-maxage=300on 2xx, short TTLs on errors, plusCDN-Cache-Controlheader.ListFeedDigestRPC aggregates ALL feeds for a variant into a single Redis-cached response. Client makes 1 request instead of ~90 per cycle. Includes circuit breaker with persistent cache fallback.Key changes
api/rss-proxy.jsproto/.../list_feed_digest.protosourcefield on ThreatClassificationserver/.../list-feed-digest.tsserver/.../_feeds.tsserver/.../_classifier.tssrc/app/data-loader.tssrc/services/ai-classify-queue.tssrc/config/variants/base.tsEstimated impact
Test plan
tsc --noEmit— 0 errorsroute-cache-tier.test.mjs— 3/4 pass (1 pre-existing:list-iran-eventsmissing)edge-functions.test.mjs— 28/28 pass/api/news/v1/list-feed-digestrequest in Network tabrss-proxygoes from ~0% to >90% in Vercel Analytics