fix(feeds): round 2 — Blockworks via Google News, plus 2 newly-surfaced dead feeds#3715
Conversation
…ed dead feeds Production console still showed errors after #3712: - Blockworks: my #3712 fix added blockworks.co to RELAY_ONLY_DOMAINS, but Cloudflare blocks BOTH Vercel edge AND Railway egress (confirmed by curl through the production proxy: still 403). Falling back to site-scoped Google News, same pattern as Kitco / Mining Weekly / FX Empire from #3712. Removed blockworks.co from RELAY_ONLY_DOMAINS — no longer reached. - Commodity Trade Mantra: direct curl returns 403 wholesale (not Vercel- specific). Same automated-traffic block as Mining Weekly. Google News fallback. - Mining Journal: migrated to Kreatio CMS; /feed/ now returns text/html (the homepage SPA), not RSS — which produced "Parse error for Mining Journal" in the client console. Google News fallback. All three Google News URLs verified to return application/xml RSS. Note on #3712: I claimed adding blockworks.co to RELAY_ONLY_DOMAINS would fix it, based on the pattern for scmp.com / iaea.org / etc. I should have probed Railway's actual response before claiming the fix would work; Blockworks is in a different Cloudflare-block tier than those domains. This PR is the actual fix.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryFixes three broken production feeds by replacing direct RSS URLs that are blocked or defunct with equivalent Google News
Confidence Score: 4/5Safe to merge — all three feed replacements follow established patterns in the codebase, The changes are minimal, well-commented, and consistent with the Google News fallback pattern already used for Kitco, Mining Weekly, FX Empire, and others. The only open item is that three now-unused domains remain in
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Feed Request] --> B{Direct RSS fetchable?}
B -- Yes --> C[Direct fetch via rss-proxy.js]
B -- No: Vercel-blocked --> D{Railway relay works?}
D -- Yes --> E[RELAY_ONLY_DOMAINS: fetch via Railway]
D -- No: both ASNs blocked --> F[Google News site: query]
B -- No: feed returns HTML/gone --> F
B -- No: wholesale 403 all IPs --> F
F --> G[news.google.com/rss/search?q=site:...]
subgraph "This PR"
H["blockworks.co → site:blockworks.co+when:1d"]
I["commoditytrademantra.com → site:commoditytrademantra.com+when:2d"]
J["mining-journal.com → site:mining-journal.com+when:2d"]
end
F --> H & I & J
Reviews (1): Last reviewed commit: "fix(feeds): round 2 — Blockworks via Goo..." | Re-trigger Greptile |
| // proxy still 403'd because Railway ASN is also blocked). Fall back to | ||
| // site-scoped Google News, matching the Kitco/Mining Weekly/FX Empire | ||
| // pattern from the same PR. | ||
| { name: 'Blockworks', url: rss('https://news.google.com/rss/search?q=site:blockworks.co+(crypto+OR+DeFi+OR+bitcoin+OR+stablecoin)+when:1d&hl=en-US&gl=US&ceid=US:en') }, |
There was a problem hiding this comment.
Stale allowed-domains entries not cleaned up
blockworks.co, www.commoditytrademantra.com, and www.mining-journal.com are still present in shared/rss-allowed-domains.json (lines 283, 287, 297) even though none of these URLs are fetched through the proxy anymore. The allowed-domains list gates which hostnames the RSS proxy will accept, so leaving defunct entries there slowly erodes its value as an authoritative record. Since all three feeds now route through news.google.com, the three stale entries could be removed in a follow-up.
…aceholders (#3717) * fix(feeds): address #3715 review — server parity, 0-item placeholders, parity guard Reviewer P1 #1: I only updated the CLIENT feed config in #3715; server-side _feeds.ts:263 still pointed Blockworks at the dead blockworks.co/feed, so the digest path (loadNewsCategory → /api/news/v1/list-feed-digest) kept hitting the Cloudflare-blocked upstream and caching zero items. Same mirror-drift class as PR #3712's allowlist miss; recorded in feedback_implement_every_site_my_own_diagnosis_enumerated. Reviewer P1 #2: I validated with `grep -c '<item>'` (counts LINES, not occurrences) and called the URLs "fine". Re-counted with `grep -o '<item>' | wc -l`: - Kitco News (50), Kitco Gold (50), Mining Weekly (100), FX Empire Gold (76), Mining Journal (19) — all real volume, fine. - Blockworks: 0 items across every probe (site: + time-window variants). Google doesn't index blockworks.co for News (Cloudflare likely blocks Googlebot on the same wholesale tier). - Commodity Trade Mantra: 0 items via site:, 2 items via "" / 30d. Effectively unindexed. Fix: REMOVE both feeds rather than ship silent placeholders. - Blockworks (client + server): The Block already exists on both sides and covers the same institutional-crypto territory. - Commodity Trade Mantra (client): commodity-news still has Mining.com / Bloomberg / Reuters / S&P Global / CNBC — coverage isn't lost. New parity test (tests/feeds-client-server-parity.test.mjs) fails when a feed NAME appears on both sides with inconsistent routing (one side Google News, the other direct). It exposed 12 PRE-EXISTING drifts (each its own per-feed judgment) — grandfathered as KNOWN_DRIFTS so this PR doesn't block on unrelated tech debt. New drift fails the test. Locked-in regression: neither file may re-add `https://blockworks.co/feed`. * fix(feeds): remove server-side Commodity Trade Mantra + extend parity test Reviewer caught a P1 on #3717: I removed Commodity Trade Mantra from the CLIENT in the first commit of this PR but missed the SERVER copy at server/worldmonitor/news/v1/_feeds.ts:341. The failure mode the reviewer described is real: 1. Server fetches 7 feeds for commodity-news, including CTM 2. Server ranks by importanceScore, then .slice(0, MAX_ITEMS_PER_CATEGORY) (list-feed-digest.ts:1076-1082) — CTM items COUNT toward the cap 3. Client filters by `enabledNames` (data-loader.ts:908-914) — CTM not in client config → CTM items DROPPED 4. Net: invisible items crowd out visible ones, shrinking the panel Two changes: 1. Server entry removed with a comment naming the exact failure mode. 2. tests/feeds-client-server-parity.test.mjs extended with: - Fixed extractor: the previous single-line regex missed ~46 multiline locale-keyed entries on the client side (France 24, EuroNews, DW News, etc.), which would have made the new orphan check fire false positives. New extractor scans up to 600 chars forward from each `name:` for the matching `url:`, capturing locale-object URLs intact. - New test: `no NEW server-only feed entries`. Server-only orphans are the exact crowd-out failure mode the reviewer caught. Five existing orphans grandfathered (Trump-Truth-Social, White House Actions, First Round Review, YC News, YC Blog) — each should be reviewed and either mirrored on the client OR removed from server. Set should SHRINK. - New regression: CTM specifically must not reappear on the server. - DW News added to KNOWN_DRIFTS: client uses mixed direct/Google-News per locale, server uses pure direct; classifier treats any-locale-GN as GN. Worth reconciling separately, not in this PR. All 5 parity tests pass; typecheck + biome clean.
Why
Production console still showed errors after #3712 (merged). Diagnostic output the operator pasted exposed three things:
blockworks.cotoRELAY_ONLY_DOMAINS, but curling the production proxy still returns 403. Cloudflare on blockworks.co blocks BOTH Vercel edge AND Railway egress (their block tier is stricter than scmp.com / iaea.org / cnn.com, which is what I extrapolated the relay fix from).commoditytrademantra.com/feed/— 503 via proxy because direct curl returns 403 wholesale. Fresh failure.mining-journal.com/feed/— 200 BUT returnstext/html(Kreatio CMS homepage), not RSS. Produced the "Parse error for Mining Journal" line in the client.Fix
site:blockworks.co+(crypto+OR+DeFi+OR+bitcoin+OR+stablecoin)+when:1dsite:commoditytrademantra.com+when:2dsite:mining-journal.com+when:2dAlso removed
blockworks.cofromRELAY_ONLY_DOMAINSinapi/rss-proxy.js— no longer reached since we're using Google News instead.All three Google News URLs verified to return
application/xml.Honest note on #3712
Adding
blockworks.cotoRELAY_ONLY_DOMAINSin #3712 was a guess based on the pattern for scmp.com / iaea.org / etc. I should have probed Railway's actual response through the production proxy before claiming the fix would work. The operator's diagnostic caught it.Verification
npm run typecheck✓npm run lint:api-contract✓api/rss-proxy.js✓npm run test:data— 8853/8853 ✓curl-verified to return real RSS (application/xml)Open from earlier diagnosis (not in this PR)
Daily Market Brief stuck on "Loading…" — still unconfirmed root cause; awaiting Network-tab data on
/api/news/v1/summarize-articlefrom a stuck panel to target precisely vs. doing a defensive shotgun fix.