fix(feeds): #3715 review — server parity + drop 0-item Google News placeholders#3717
Conversation
…, parity guard Reviewer P1 #1: I only updated the CLIENT feed config in #3715; server-side _feeds.ts:263 still pointed Blockworks at the dead blockworks.co/feed, so the digest path (loadNewsCategory → /api/news/v1/list-feed-digest) kept hitting the Cloudflare-blocked upstream and caching zero items. Same mirror-drift class as PR #3712's allowlist miss; recorded in feedback_implement_every_site_my_own_diagnosis_enumerated. Reviewer P1 #2: I validated with `grep -c '<item>'` (counts LINES, not occurrences) and called the URLs "fine". Re-counted with `grep -o '<item>' | wc -l`: - Kitco News (50), Kitco Gold (50), Mining Weekly (100), FX Empire Gold (76), Mining Journal (19) — all real volume, fine. - Blockworks: 0 items across every probe (site: + time-window variants). Google doesn't index blockworks.co for News (Cloudflare likely blocks Googlebot on the same wholesale tier). - Commodity Trade Mantra: 0 items via site:, 2 items via "" / 30d. Effectively unindexed. Fix: REMOVE both feeds rather than ship silent placeholders. - Blockworks (client + server): The Block already exists on both sides and covers the same institutional-crypto territory. - Commodity Trade Mantra (client): commodity-news still has Mining.com / Bloomberg / Reuters / S&P Global / CNBC — coverage isn't lost. New parity test (tests/feeds-client-server-parity.test.mjs) fails when a feed NAME appears on both sides with inconsistent routing (one side Google News, the other direct). It exposed 12 PRE-EXISTING drifts (each its own per-feed judgment) — grandfathered as KNOWN_DRIFTS so this PR doesn't block on unrelated tech debt. New drift fails the test. Locked-in regression: neither file may re-add `https://blockworks.co/feed`.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR completes the cleanup started in #3715 by removing Blockworks from the server-side feed config (fixing the client/server mirror drift), dropping two 0-item Google News placeholders (Blockworks and Commodity Trade Mantra on the client), and adding a new parity regression test to prevent the same class of drift from recurring.
Confidence Score: 4/5Safe to merge — the config removals are straightforward and well-justified; the new test adds coverage with minor blind spots that do not affect correctness today. The server-side Commodity Trade Mantra entry now has no client counterpart, silently dropping it from the parity test monitoring scope. The name is also misleading — the underlying gn() query is a generic topic search, not a publisher-specific one. Neither issue causes wrong data today, but the orphaned entry is a gap in the regression net this PR specifically set out to build. server/worldmonitor/news/v1/_feeds.ts line 341 — the Commodity Trade Mantra entry is now server-only and uses a generic query that does not reflect its name; worth renaming before a future contributor mistakes it for a publisher-specific feed. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[extractFeedRouting client feeds.ts] --> C[client Map name to isGoogleNews]
B[extractFeedRouting server _feeds.ts] --> D[server Map name to isGoogleNews]
C --> E{Shared name on both sides?}
D --> E
E -- No --> F[Silently skipped e.g. Commodity Trade Mantra after this PR]
E -- Yes --> G{isGoogleNews consistent?}
G -- Consistent + in KNOWN_DRIFTS --> H[resolvedKnown list - fail: remove from set]
G -- Consistent --> I[Pass]
G -- Inconsistent + in KNOWN_DRIFTS --> J[Grandfathered]
G -- Inconsistent + NEW --> K[newDrift list - fail: fix or rename]
subgraph Regression lock
L[Blockworks regex check blockworks.co/feed absent on both sides]
end
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… test Reviewer caught a P1 on #3717: I removed Commodity Trade Mantra from the CLIENT in the first commit of this PR but missed the SERVER copy at server/worldmonitor/news/v1/_feeds.ts:341. The failure mode the reviewer described is real: 1. Server fetches 7 feeds for commodity-news, including CTM 2. Server ranks by importanceScore, then .slice(0, MAX_ITEMS_PER_CATEGORY) (list-feed-digest.ts:1076-1082) — CTM items COUNT toward the cap 3. Client filters by `enabledNames` (data-loader.ts:908-914) — CTM not in client config → CTM items DROPPED 4. Net: invisible items crowd out visible ones, shrinking the panel Two changes: 1. Server entry removed with a comment naming the exact failure mode. 2. tests/feeds-client-server-parity.test.mjs extended with: - Fixed extractor: the previous single-line regex missed ~46 multiline locale-keyed entries on the client side (France 24, EuroNews, DW News, etc.), which would have made the new orphan check fire false positives. New extractor scans up to 600 chars forward from each `name:` for the matching `url:`, capturing locale-object URLs intact. - New test: `no NEW server-only feed entries`. Server-only orphans are the exact crowd-out failure mode the reviewer caught. Five existing orphans grandfathered (Trump-Truth-Social, White House Actions, First Round Review, YC News, YC Blog) — each should be reviewed and either mirrored on the client OR removed from server. Set should SHRINK. - New regression: CTM specifically must not reappear on the server. - DW News added to KNOWN_DRIFTS: client uses mixed direct/Google-News per locale, server uses pure direct; classifier treats any-locale-GN as GN. Worth reconciling separately, not in this PR. All 5 parity tests pass; typecheck + biome clean.
|
Closing the loop on Greptile's review — Greptile only reviewed commit
Both fixes verified live on main ( |
1 participant
Addresses the two P1s left on PR #3715 review after merge.
P1 #1 — Server-side mirror drift (Blockworks)
In #3715 I removed
blockworks.co/feedfromsrc/config/feeds.ts(the client side) but left it intact on the server side atserver/worldmonitor/news/v1/_feeds.ts:263. The Finance web-digest path (loadNewsCategory→/api/news/v1/list-feed-digest) keeps hitting the Cloudflare-blocked upstream and caching zero items.Fix: remove the same entry from
_feeds.tswith a comment explaining the parity. The Block (still present on both sides) covers institutional crypto territory — no coverage lost.Same mirror-drift class as PR #3712's allowlist miss; recorded in
feedback_implement_every_site_my_own_diagnosis_enumerated.P1 #2 — 0-item Google News placeholders
When I validated the Google News replacement URLs in #3715, I used `grep -c ''` — which counts LINES, not occurrences, and the RSS bodies are mostly single-line. Re-counted with `grep -o '' | wc -l`:
The first five are fine. The two zero-count ones are silent placeholders — Google effectively doesn't index those domains for News (publisher likely blocks Googlebot on the same Cloudflare wholesale tier that blocks Vercel + Railway).
Fix: drop both rather than ship empty-result feeds.
Guard against future mirror drift
`tests/feeds-client-server-parity.test.mjs` walks both feed configs, extracts `{ name, routing }` for every entry, and fails when a shared name routes inconsistently (one side Google News, the other a direct fetch). It found 12 pre-existing drifts at this commit; those are grandfathered as `KNOWN_DRIFTS` (each is its own per-feed judgment and should be cleaned up incrementally — shrinking the set, never growing it). New drift fails the test.
A locked-in regression test also fails if either file re-introduces `https://blockworks.co/feed\`.
Test plan