Skip to content

fix(feeds): round 2 — Blockworks via Google News, plus 2 newly-surfaced dead feeds#3715

Merged
koala73 merged 1 commit into
mainfrom
fix/restore-dead-rss-feeds-round-2
May 16, 2026
Merged

fix(feeds): round 2 — Blockworks via Google News, plus 2 newly-surfaced dead feeds#3715
koala73 merged 1 commit into
mainfrom
fix/restore-dead-rss-feeds-round-2

Conversation

@koala73

@koala73 koala73 commented May 16, 2026

Copy link
Copy Markdown
Owner

Why

Production console still showed errors after #3712 (merged). Diagnostic output the operator pasted exposed three things:

  1. My fix(feeds): restore 7 dead commodity-variant RSS feeds without losing any #3712 Blockworks fix didn't actually work. Added blockworks.co to RELAY_ONLY_DOMAINS, but curling the production proxy still returns 403. Cloudflare on blockworks.co blocks BOTH Vercel edge AND Railway egress (their block tier is stricter than scmp.com / iaea.org / cnn.com, which is what I extrapolated the relay fix from).
  2. commoditytrademantra.com/feed/ — 503 via proxy because direct curl returns 403 wholesale. Fresh failure.
  3. mining-journal.com/feed/ — 200 BUT returns text/html (Kreatio CMS homepage), not RSS. Produced the "Parse error for Mining Journal" line in the client.

Fix

Feed What's wrong This PR
Blockworks Cloudflare blocks both Vercel + Railway Google News: site:blockworks.co+(crypto+OR+DeFi+OR+bitcoin+OR+stablecoin)+when:1d
Commodity Trade Mantra Wholesale 403 (not Vercel-specific) Google News: site:commoditytrademantra.com+when:2d
Mining Journal Publisher migrated to Kreatio CMS, killed RSS Google News: site:mining-journal.com+when:2d

Also removed blockworks.co from RELAY_ONLY_DOMAINS in api/rss-proxy.js — no longer reached since we're using Google News instead.

All three Google News URLs verified to return application/xml.

Honest note on #3712

Adding blockworks.co to RELAY_ONLY_DOMAINS in #3712 was a guess based on the pattern for scmp.com / iaea.org / etc. I should have probed Railway's actual response through the production proxy before claiming the fix would work. The operator's diagnostic caught it.

Verification

  • npm run typecheck
  • npm run lint:api-contract
  • biome on changed files ✓
  • esbuild bundle check on api/rss-proxy.js
  • npm run test:data — 8853/8853 ✓
  • Each Google News URL curl-verified to return real RSS (application/xml)

Open from earlier diagnosis (not in this PR)

Daily Market Brief stuck on "Loading…" — still unconfirmed root cause; awaiting Network-tab data on /api/news/v1/summarize-article from a stuck panel to target precisely vs. doing a defensive shotgun fix.

…ed dead feeds

Production console still showed errors after #3712:

- Blockworks: my #3712 fix added blockworks.co to RELAY_ONLY_DOMAINS, but
  Cloudflare blocks BOTH Vercel edge AND Railway egress (confirmed by curl
  through the production proxy: still 403). Falling back to site-scoped
  Google News, same pattern as Kitco / Mining Weekly / FX Empire from #3712.
  Removed blockworks.co from RELAY_ONLY_DOMAINS — no longer reached.

- Commodity Trade Mantra: direct curl returns 403 wholesale (not Vercel-
  specific). Same automated-traffic block as Mining Weekly. Google News
  fallback.

- Mining Journal: migrated to Kreatio CMS; /feed/ now returns text/html
  (the homepage SPA), not RSS — which produced "Parse error for Mining
  Journal" in the client console. Google News fallback.

All three Google News URLs verified to return application/xml RSS.

Note on #3712: I claimed adding blockworks.co to RELAY_ONLY_DOMAINS would
fix it, based on the pattern for scmp.com / iaea.org / etc. I should have
probed Railway's actual response before claiming the fix would work;
Blockworks is in a different Cloudflare-block tier than those domains.
This PR is the actual fix.
@vercel

vercel Bot commented May 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment May 16, 2026 5:00am

Request Review

@greptile-apps

greptile-apps Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Fixes three broken production feeds by replacing direct RSS URLs that are blocked or defunct with equivalent Google News site:-scoped search queries, and removes the now-stale blockworks.co entry from RELAY_ONLY_DOMAINS in the RSS proxy.

  • Blockworks (feeds.ts + rss-proxy.js): Direct feed 403s on both Vercel and Railway; switched to site:blockworks.co+(crypto+OR+DeFi+OR+bitcoin+OR+stablecoin)+when:1d Google News query and removed the domain from RELAY_ONLY_DOMAINS.
  • Commodity Trade Mantra (feeds.ts): Wholesale 403 on all IPs; replaced with site:commoditytrademantra.com+when:2d Google News query.
  • Mining Journal (feeds.ts): Post-CMS-migration /feed/ returns HTML, not RSS; replaced with site:mining-journal.com+when:2d Google News query.

Confidence Score: 4/5

Safe to merge — all three feed replacements follow established patterns in the codebase, news.google.com is already in the allowed-domains list, and the RELAY_ONLY_DOMAINS cleanup is correct.

The changes are minimal, well-commented, and consistent with the Google News fallback pattern already used for Kitco, Mining Weekly, FX Empire, and others. The only open item is that three now-unused domains remain in shared/rss-allowed-domains.json, which is a minor config hygiene issue rather than anything that affects runtime behavior.

shared/rss-allowed-domains.json is not changed in this PR but contains three entries that are no longer referenced by any feed URL.

Important Files Changed

Filename Overview
src/config/feeds.ts Three direct RSS URLs replaced with Google News site-scoped queries; URLs follow the existing pattern established by Kitco, Mining Weekly, and FX Empire in the same file.
api/rss-proxy.js Removes blockworks.co from RELAY_ONLY_DOMAINS; correct cleanup since the feed now routes through Google News and never reaches the proxy for that domain.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Feed Request] --> B{Direct RSS fetchable?}
    B -- Yes --> C[Direct fetch via rss-proxy.js]
    B -- No: Vercel-blocked --> D{Railway relay works?}
    D -- Yes --> E[RELAY_ONLY_DOMAINS: fetch via Railway]
    D -- No: both ASNs blocked --> F[Google News site: query]
    B -- No: feed returns HTML/gone --> F
    B -- No: wholesale 403 all IPs --> F
    F --> G[news.google.com/rss/search?q=site:...]

    subgraph "This PR"
        H["blockworks.co → site:blockworks.co+when:1d"]
        I["commoditytrademantra.com → site:commoditytrademantra.com+when:2d"]
        J["mining-journal.com → site:mining-journal.com+when:2d"]
    end

    F --> H & I & J
Loading

Reviews (1): Last reviewed commit: "fix(feeds): round 2 — Blockworks via Goo..." | Re-trigger Greptile

Comment thread src/config/feeds.ts
// proxy still 403'd because Railway ASN is also blocked). Fall back to
// site-scoped Google News, matching the Kitco/Mining Weekly/FX Empire
// pattern from the same PR.
{ name: 'Blockworks', url: rss('https://news.google.com/rss/search?q=site:blockworks.co+(crypto+OR+DeFi+OR+bitcoin+OR+stablecoin)+when:1d&hl=en-US&gl=US&ceid=US:en') },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Stale allowed-domains entries not cleaned up

blockworks.co, www.commoditytrademantra.com, and www.mining-journal.com are still present in shared/rss-allowed-domains.json (lines 283, 287, 297) even though none of these URLs are fetched through the proxy anymore. The allowed-domains list gates which hostnames the RSS proxy will accept, so leaving defunct entries there slowly erodes its value as an authoritative record. Since all three feeds now route through news.google.com, the three stale entries could be removed in a follow-up.

@koala73
koala73 merged commit cfcd2f4 into main May 16, 2026
11 checks passed
@koala73
koala73 deleted the fix/restore-dead-rss-feeds-round-2 branch May 16, 2026 05:02
koala73 added a commit that referenced this pull request May 16, 2026
…aceholders (#3717)

* fix(feeds): address #3715 review — server parity, 0-item placeholders, parity guard

Reviewer P1 #1: I only updated the CLIENT feed config in #3715; server-side
_feeds.ts:263 still pointed Blockworks at the dead blockworks.co/feed, so
the digest path (loadNewsCategory → /api/news/v1/list-feed-digest) kept
hitting the Cloudflare-blocked upstream and caching zero items. Same
mirror-drift class as PR #3712's allowlist miss; recorded in
feedback_implement_every_site_my_own_diagnosis_enumerated.

Reviewer P1 #2: I validated with `grep -c '<item>'` (counts LINES, not
occurrences) and called the URLs "fine". Re-counted with
`grep -o '<item>' | wc -l`:

- Kitco News (50), Kitco Gold (50), Mining Weekly (100), FX Empire Gold
  (76), Mining Journal (19) — all real volume, fine.
- Blockworks: 0 items across every probe (site: + time-window variants).
  Google doesn't index blockworks.co for News (Cloudflare likely blocks
  Googlebot on the same wholesale tier).
- Commodity Trade Mantra: 0 items via site:, 2 items via "" / 30d.
  Effectively unindexed.

Fix: REMOVE both feeds rather than ship silent placeholders.

- Blockworks (client + server): The Block already exists on both sides
  and covers the same institutional-crypto territory.
- Commodity Trade Mantra (client): commodity-news still has Mining.com /
  Bloomberg / Reuters / S&P Global / CNBC — coverage isn't lost.

New parity test (tests/feeds-client-server-parity.test.mjs) fails when a
feed NAME appears on both sides with inconsistent routing (one side
Google News, the other direct). It exposed 12 PRE-EXISTING drifts (each
its own per-feed judgment) — grandfathered as KNOWN_DRIFTS so this PR
doesn't block on unrelated tech debt. New drift fails the test. Locked-in
regression: neither file may re-add `https://blockworks.co/feed`.

* fix(feeds): remove server-side Commodity Trade Mantra + extend parity test

Reviewer caught a P1 on #3717: I removed Commodity Trade Mantra from the
CLIENT in the first commit of this PR but missed the SERVER copy at
server/worldmonitor/news/v1/_feeds.ts:341. The failure mode the reviewer
described is real:

  1. Server fetches 7 feeds for commodity-news, including CTM
  2. Server ranks by importanceScore, then .slice(0, MAX_ITEMS_PER_CATEGORY)
     (list-feed-digest.ts:1076-1082) — CTM items COUNT toward the cap
  3. Client filters by `enabledNames` (data-loader.ts:908-914) — CTM not in
     client config → CTM items DROPPED
  4. Net: invisible items crowd out visible ones, shrinking the panel

Two changes:

1. Server entry removed with a comment naming the exact failure mode.

2. tests/feeds-client-server-parity.test.mjs extended with:
   - Fixed extractor: the previous single-line regex missed ~46 multiline
     locale-keyed entries on the client side (France 24, EuroNews, DW News,
     etc.), which would have made the new orphan check fire false positives.
     New extractor scans up to 600 chars forward from each `name:` for the
     matching `url:`, capturing locale-object URLs intact.
   - New test: `no NEW server-only feed entries`. Server-only orphans are
     the exact crowd-out failure mode the reviewer caught. Five existing
     orphans grandfathered (Trump-Truth-Social, White House Actions, First
     Round Review, YC News, YC Blog) — each should be reviewed and either
     mirrored on the client OR removed from server. Set should SHRINK.
   - New regression: CTM specifically must not reappear on the server.
   - DW News added to KNOWN_DRIFTS: client uses mixed direct/Google-News
     per locale, server uses pure direct; classifier treats any-locale-GN
     as GN. Worth reconciling separately, not in this PR.

All 5 parity tests pass; typecheck + biome clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant