fix(brief): add source-domain STRONG signal to classifyOpinion (May 19 regression)#3835
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds a third STRONG signal to
Confidence Score: 3/5The core classifier logic and URL-parsing safety are sound, but including foreignpolicy.com in the blanket allowlist may silently suppress genuine breaking-news stories from that outlet. The Bulletin/Project Syndicate/War on the Rocks inclusions are editorially clean — those outlets truly publish no hard-news wires. Foreign Policy is materially different: it runs a continuous news desk alongside its analysis pieces, so a blanket hostname match will drop real events without any fallback. That is a present, production-visible drop in brief quality on every FP story from the moment this deploys. The COMMENTARY_HOSTNAMES Set in server/_shared/opinion-classifier.js — specifically the foreignpolicy.com entry — needs editorial verification before merge. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[classifyOpinion story] --> B[Extract title / link / description]
B --> C[safePathname link → pathname]
C --> D{STRONG 1\nURL path segment\n/opinion/, /views/, …}
D -->|match| Z[return true\nopinion]
D -->|no match| E{STRONG 2\nHeadline prefix\nOpinion: Analysis: …}
E -->|match| Z
E -->|no match| F[safeHostname link → hostname\nnew in this PR]
F --> G{STRONG 3 NEW\nmatchesCommentaryHost\nhostname === entry OR\nhostname.endsWith '.' + entry}
G -->|match| Z
G -->|no match| H[Accumulate CORROBORATING signals]
H --> I{corroborating ≥ 2?}
I -->|yes| Z
I -->|no| J[return false\nhard news]
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| // section to distinguish from. Hostname match on the parsed URL | ||
| // only, suffix-anchored to permit `newsletter.<host>` and `m.<host>` | ||
| // while rejecting typo-domains. | ||
| if (matchesCommentaryHost(safeHostname(link))) return true; |
There was a problem hiding this comment.
| if (matchesCommentaryHost(safeHostname(link))) return true; | |
| if (matchesCommentaryHost(hostname)) return true; |
| 'foreignpolicy.com', // Foreign Policy magazine | ||
| 'warontherocks.com', // War on the Rocks (defense analysis blog) |
There was a problem hiding this comment.
foreignpolicy.com may over-filter breaking-news events
Foreign Policy runs a continuous news desk alongside its analysis columns — "The Cable" and their daily news briefings cover diplomatic events, troop movements, and sanctions announcements that are structurally identical to hard-news stories you'd want in the brief. Classifying every foreignpolicy.com URL as opinion would silently drop those events. The selection criterion in the comment above ("They do NOT publish breaking-news wires or event coverage") is not met by FP the way it is for, say, thebulletin.org or project-syndicate.org. Worth confirming this was intentional before it ships, or narrowing the match to FP's known opinion subdirectories instead.
…9 regression) The 2026-05-19 Pro brief shipped "How nuclear war would impact the global food system" from Bulletin of Atomic Scientists as CRITICAL story #6, sitting alongside breaking news about the Iran-Israel war and the WHO Ebola declaration. The brief promises event-driven intelligence — a Bulletin analysis essay is not an event. The existing classifier missed it on all three current signals: - STRONG #1 (URL section /opinion/): Bulletin URLs have no opinion- style path because the WHOLE SITE is commentary, no hard-news section to distinguish from. - STRONG #2 (headline prefix "Opinion:"): hard-news-shaped title. - CORROBORATING (quote-wrap + columnist framing): description reads like a news lede. The signal IS the publisher. Add STRONG #3: a hand-curated allowlist of commentary-only publishers, matched by hostname (suffix-anchored to permit `newsletter.<host>` / `m.<host>` while rejecting typo-domains like `evilthebulletin.org`). Initial list (per docs/plans/2026-05-19-001 U1): - thebulletin.org — Bulletin of Atomic Scientists - project-syndicate.org — op-eds from world leaders / academics - foreignaffairs.com — CFR's analysis quarterly - foreignpolicy.com — Foreign Policy magazine - warontherocks.com — defense analysis blog Maintenance commitment: quarterly review against `droppedOpinion` telemetry to catch (a) new commentary publishers to add, (b) listed publishers that launched a hard-news section. Rollback path is remove-from-Set, never per-URL exceptions (cruft). Pattern mirrors the existing safePathname/STRONG_URL_SEGMENTS shape; new safeHostname helper parses URL().hostname so the match is on the parsed host (not raw .includes() on the link string — closes the tracking-param injection vector documented in PR #3748). 7 new test cases cover the May 19 regression, the 4 other publishers, subdomain matching, typo-domain rejection, tracking-param / fragment spoofing, malformed URLs, and the "allowlisted host PLUS hard-news content → still opinion" rule. This is PR-1 of the 4-PR Phase 1 wave from docs/plans/2026-05-19-001-fix-brief-content-quality-regressions-plan.md.
48c0696 to
c32fbcf
Compare
|
P1 confirmed and fixed in Foreign Policy runs hard-news surfaces (World Brief, Situation Report, Morning Brief) alongside their op-ed section — blanket hostname allowlist violates the file's own selection criterion #2 ("they do NOT publish breaking-news / event coverage"). Your verbatim G-7 finance-ministers story would have been silently dropped. Changes:
Verified the other 4 hostnames still meet criterion #2:
27 tests pass (was 25); 2 new regression cases pin the FP exclusion against future re-addition. |
1 participant
Symptom
The 2026-05-19 0801 Pro brief shipped "How nuclear war would impact the global food system" from Bulletin of Atomic Scientists as CRITICAL story #6, sitting alongside breaking news about the Iran-Israel war and the WHO Ebola declaration. Pro subscriber filed the complaint that surfaced 5 distinct content-quality defects on the same day; this PR addresses defect #2.
Root cause
`classifyOpinion` has two existing STRONG signals (URL path segments + headline prefix) and three CORROBORATING signals (description framing, quote-wrap, `/analysis/` path). All five missed the Bulletin piece:
The signal IS the publisher. Bulletin of Atomic Scientists, Project Syndicate, Foreign Affairs, etc. are entirely commentary — no hard-news section to discriminate via URL path.
Fix
Add STRONG #3: a hand-curated hostname allowlist matched against the parsed `URL().hostname` (not raw `.includes()`), suffix-anchored so subdomains (`newsletter.`, `m.`, `www.`) match but typo-domains (`evilthebulletin.org`) don't.
Initial list (per docs/plans/2026-05-19-001 U1):
Maintenance commitment documented in the source: quarterly review against `droppedOpinion` telemetry. Rollback is remove-from-Set, never ad-hoc URL exceptions.
Test plan
Origin
PR-1 of the 4-PR Phase 1 wave from `docs/plans/2026-05-19-001-fix-brief-content-quality-regressions-plan.md`.
Sibling pattern: PR #3690 (opinion classifier original) and the May 17 feelgood-filter plan use the same ingest-stamp + read-time-re-classify wiring. This PR adds a new STRONG signal to the existing classifier; no new classifier, no new ingest-side stamp.