fix(brief): exclude opinion/analysis from the pool + lead↔card-#1 coherence telemetry#3690
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds two targeted fixes to the brief pipeline: an opinion/analysis exclusion filter (F3) and lead ↔ card-#1 coherence telemetry (F4). Both address concrete May 14 production incidents where an op-ed column ranked above a nuclear test and the synthesized lead described a different story than the rendered first card.
Confidence Score: 4/5Safe to merge; both features are well-scoped and the F4 path is measure-only with no brief held back. The two-layer opinion filter and the coherence telemetry are straightforward additions with solid unit-test coverage. The only open question is whether server/_shared/opinion-classifier.js — specifically the Important Files Changed
Reviews (1): Last reviewed commit: "fix(brief): exclude opinion/analysis fro..." | Re-trigger Greptile |
| const STRONG_URL_SEGMENTS = [ | ||
| '/opinion/', | ||
| '/opinions/', | ||
| '/opinion-', | ||
| '/views/', |
There was a problem hiding this comment.
The
/opinion- prefix segment matches any URL path that starts with opinion-, which can false-positive on election-polling or survey URLs like /opinion-polls/monthly or /opinion-leaders/. The PR's own design principle calls false positives (silently dropped real events) more harmful than false negatives. The existing entries /opinion/ and /opinions/ already cover the standard opinion-section pattern; the /opinion- suffix covers cases like /opinion-editorial/, but those could be listed explicitly instead.
| const STRONG_URL_SEGMENTS = [ | |
| '/opinion/', | |
| '/opinions/', | |
| '/opinion-', | |
| '/views/', | |
| const STRONG_URL_SEGMENTS = [ | |
| '/opinion/', | |
| '/opinions/', | |
| '/opinion-editorial/', | |
| '/opinion-comment/', | |
| '/views/', |
There was a problem hiding this comment.
Already addressed in ea2645c (this review was on b065082, one commit behind). /opinion- was removed from STRONG_URL_SEGMENTS entirely — it was an unbounded substring that false-positived on hard-news slugs like /world/opinion-polls-tighten-election. Every remaining entry is slash-delimited on both sides. A regression test for the exact opinion-polls slug was added to tests/opinion-classifier.test.mjs.
…on hard-news slugs PR #3690 review catch. STRONG_URL_SEGMENTS included `/opinion-` (no closing slash) as a substring match — it caught hard-news ARTICLE SLUGS like `/world/opinion-polls-tighten-election`, classifying real election/polling coverage as opinion. Since buildDigest drops rows where classifyOpinion() is true, that silently removed those stories from briefs. Every other STRONG_URL_SEGMENTS entry is slash-delimited on both sides — a real path segment. `/opinion-` was the lone unbounded prefix. Removed it. Genuine opinion sections (`/opinion/`, `/opinions/`, `/commentary/`, `/editorial/`, `/op-ed/`, `/columnists/`, `/columns/`) are still caught. My original test had the gap that let this through: it checked the "Opinion polls tighten" HEADLINE with link `/world/polls` — never the `/world/opinion-polls-…` URL form. Added a regression test for the exact slug, plus a counter-control confirming a genuine `/opinion/` section still classifies. 12/12 classifier tests, biome clean, typecheck clean.
…erence telemetry PR C of the brief-pipeline plan (Phase 3 F3 + Phase 4 F4). Stacked on PR B (#3688). ── F3: opinion/analysis exclusion ─────────────────────────────────── The May 14 brief ranked a Le Monde opinion column ("'Russia's invasion of Ukraine could have warned Trump…'", by columnist Gilles Paris) as story #1, tagged Critical, ahead of a nuclear ICBM test. The brief is event-driven intelligence — an op-ed column is not an event. - New `server/_shared/opinion-classifier.js` — single shared classifier `classifyOpinion({ title, link, description })`. Tiered: STRONG signals (URL /opinion//views//commentary//editorial//op-ed/ /columnists/ segments; explicit Opinion:/Analysis:/Commentary:/ Op-Ed: headline prefix) classify alone; CORROBORATING signals (whole-headline quote-wrap, description framing words, /analysis/ URL) need a STRONG signal OR two corroborating. Conservative — a false negative ships one op-ed; a false positive silently drops a real event. Note: the plan expected ingest-time byline/section metadata, but the parsed RSS item and the story:track:v1 row carry neither — only title, link, description. Both layers classify from the same three signals. - Two-layer filter: · Ingest (list-feed-digest.ts) — ParsedItem gains `isOpinion`, set by classifyOpinion in parseRssXml; buildStoryTrackHsetFields stamps `isOpinion` ('1'|'0') on the story:track:v1 row. · Read-time (buildDigest) — drops rows where `isOpinion === '1'`; for residue rows ingested BEFORE the stamp shipped (no isOpinion field), re-classifies from the persisted title/link/description. Emits `[digest] buildDigest opinion filter dropped N` telemetry. Stamps rather than drops at ingest — story:track rows feed more than the brief; only buildDigest (the brief's read path) filters. ── F4: lead ↔ final-card-#1 coherence ─────────────────────────────── The synthesis emits `lead` and `rankedStoryHashes` independently, with no constraint that the lead is ABOUT the rendered first story. And `rankedStoryHashes[0]` ≠ `data.stories[0]` — orderBriefCandidates re-sorts by severity/topic/score after the LLM rank. On May 14 the lead was about the Ukraine-energy story while data.stories[0] was the Le Monde column. - New `leadGroundsAgainstStory(lead, headline)` in brief-llm.mjs — reuses the checkLeadGrounding anchor machinery (capitalised, ≥4 chars, stopword-filtered headline anchors; token-set membership) but with a FIXED threshold of 1. checkLeadGrounding is the wrong fit: scoped to one story, a single headline can carry ≥4 anchors, tripping its size-based threshold up to 2 — too strict for a coherence question ("same story?", not "how grounded?"). - `composeAndStoreBriefForUser` runs the check AFTER composeBriefFromDigestStories, against `envelope.data.stories[0]` (the final ordered first card) — NEVER `rankedStoryHashes[0]`. It runs in the orchestration layer, not the pure composer, so composeBriefFromDigestStories stays I/O-free. Measure-first (plan F4 option b): emits `[digest] lead card1 coherence` every brief + a `LEAD/CARD-#1 INCOHERENCE` warn on mismatch; ships the brief as-is. Once the mismatch rate is known, decide between regenerating the lead bound to stories[0] or a separate leadStoryHash. ── Tests ──────────────────────────────────────────────────────────── - 11 classifyOpinion unit tests: STRONG signals alone, two- corroborating, one-corroborating-NOT-enough, the verbatim May 14 Le Monde column → opinion, /analysis/ hard-news NOT dropped, quoted-PHRASE (not whole-wrap) hard-news NOT dropped, bare-noun "Opinion polls…" NOT caught, input safety. - parseRssXml carries isOpinion (hard-news false / op-ed true). - buildStoryTrackHsetFields stamps isOpinion '1'/'0' + legacy missing-field → '0'. - 3 leadGroundsAgainstStory tests incl. the May 14 F4 regression (lead about Netanyahu vs Le Monde card-#1 → incoherent) and the degenerate no-anchor → skip. - Dockerfile.digest-notifications COPYs the new opinion-classifier.js (the transitive-import-closure test caught the missing COPY). 8728/8728 full unit suite, typecheck clean (both configs), biome clean (3 pre-existing warnings: buildDigest/main complexity already 5x over limit, one useOptionalChain on untouched list-feed-digest code — none introduced by this PR). Note: no cache-prefix bump needed — F3 changes the digest POOL membership (fewer stories), not the synthesis cache material; F4 is pure telemetry.
…on hard-news slugs PR #3690 review catch. STRONG_URL_SEGMENTS included `/opinion-` (no closing slash) as a substring match — it caught hard-news ARTICLE SLUGS like `/world/opinion-polls-tighten-election`, classifying real election/polling coverage as opinion. Since buildDigest drops rows where classifyOpinion() is true, that silently removed those stories from briefs. Every other STRONG_URL_SEGMENTS entry is slash-delimited on both sides — a real path segment. `/opinion-` was the lone unbounded prefix. Removed it. Genuine opinion sections (`/opinion/`, `/opinions/`, `/commentary/`, `/editorial/`, `/op-ed/`, `/columnists/`, `/columns/`) are still caught. My original test had the gap that let this through: it checked the "Opinion polls tighten" HEADLINE with link `/world/polls` — never the `/world/opinion-polls-…` URL form. Added a regression test for the exact slug, plus a counter-control confirming a genuine `/opinion/` section still classifies. 12/12 classifier tests, biome clean, typecheck clean.
ea2645c to
de7921f
Compare
…ult leakage PR-review finding: pre-PR cached ParseResults in rss:feed:v3 contain ParsedItems without isFeelGood. If a cache hit returned one during the 1h healthy-cache rollout window, buildStoryTrackHsetFields would write `'isFeelGood', undefined ? '1' : '0'` → '0' onto the fresh story:track:v1 row. buildDigest's stampMissing check (`typeof !== 'string' || length === 0`) treats '0' as a genuine "not feel-good" verdict and skips the residue catch — feel-good content could silently slip through for up to one hour post-deploy. Fix: bump the cache prefix v3→v4 so every pre-PR ParseResult is invalidated on deploy. The first post-deploy cron tick is a cold read for every feed; fresh parseRssXml runs stamp isFeelGood correctly before buildStoryTrackHsetFields ever sees the item. Mirrors the v2→v3 precedent on this same cache for the same class of parsed-cache-shape change. Same latent bug exists in PR #3690's isOpinion path (1h leakage window after that PR shipped, masked by the TTL). A separate backport can address it; out of scope for this PR. New test: tests/news-story-track-description-persistence.test.mts asserts the v4 cache prefix is present in fetchAndParseRss source — locks the cutover so a refactor cannot silently revert to v3. Updated comment on the existing missing-field test to clarify that the '0' fallback is buildStoryTrackHsetFields' own behavior; the cache-leakage failure mode is closed by the v4 prefix bump above.
…3748) * feat(brief): add feelgood-classifier shared module (U1) Sibling to classifyOpinion (PR #3690). Stamps/filters editorially-unfit feel-good and lifestyle content out of the brief pool. Design (round-2 ce-doc-review): - HARD-NEWS VETO (R3a) runs FIRST; overrides every classification path (STRONG URL, STRONG headline prefix, CORROBORATING). Expanded with morphology variants so natural news prose vetoes correctly ("strike kills six" — not requiring exact "airstrike" / "killed"). - CORROBORATING threshold raised from 2 to 3 distinct group names (adv-R2-003 — prevents reachable hard-news FPs on 2-token stacks). - Distinct-token identity is the alternation-group LABEL, not raw match text (adv-R2-002 — collapses reunite/reunited/reuniting into reunite_group so RSS-typical inflection echoes count once). - /local/, /photos/, /travel/, /style/ are CORROBORATING only — major outlets file legitimate hard news under those segments (adv-R2-001). - URL match uses new URL().pathname (try/catch), not raw .includes() — closes the tracking-param injection vector (?utm=/local/promo). - "restored" and "meet the" excluded from token list (FPs on art-restitution news and diplomacy headlines). See docs/plans/2026-05-17-001-fix-feelgood-lifestyle-filter-plan.md for full design rationale + Veterans-warplanes anchor case (May 17 0802 brief, card #4). 36 tests; biome clean. * feat(brief): stamp isFeelGood on story:track:v1 at ingest (U2) Sibling to the isOpinion stamp PR #3690 added. parseRssXml computes classifyFeelGood({title, link, description}) and tags every ParsedItem; buildStoryTrackHsetFields persists 'isFeelGood' as '1'|'0' on the Redis HSET row so buildDigest's read-path filter can exclude feel-good content without re-classifying. Mirrors the four isOpinion sites exactly: - Import at :17 (sibling to opinion-classifier import) - ParsedItem field at :163 (sibling to isOpinion) - parseRssXml stamp at :478 (sibling to classifyOpinion call) - buildStoryTrackHsetFields persist at :911 (sibling to isOpinion field) Tests: 2 new (parseRssXml carries the flag + Veterans anchor; HSET persistence + legacy-residue → '0'). Existing baseItem fixture gains isFeelGood: false default (matches PR #3690's isOpinion pattern). * feat(brief): drop isFeelGood rows in buildDigest + telemetry (U3) Sibling to the buildDigest opinion filter PR #3690 added. Trusts the ingest stamp (isFeelGood === "1") AND re-classifies stamp-missing residue rows from persisted title/link/description so the filter is effective immediately on rollout, not only after the 48h TTL window. Mirrors the four opinion filter sites exactly: - Import classifyFeelGood at :34 (sibling to opinion-classifier import) - droppedFeelGood counter at :490 (sibling to droppedOpinion) - Filter block immediately after opinion block (:522-549), BEFORE derivePhase / matchesSensitivity — earlier filtering keeps downstream telemetry clean - Conditional log at :567-573 mirrors the opinion log byte-for-byte: "[digest] buildDigest feel-good filter dropped N feel-good/lifestyle item(s) from the pool (variant=… lang=… sensitivity=…)". Per FEAS-001, no invented stamped/residue parenthetical (opinion mirror has none). M6 / adv-005 — counter asymmetry documented in inline comment: a row matched by BOTH classifiers (columnist-nostalgia essay; op-ed with tribute framing) increments only droppedOpinion (opinion `continue`s first). droppedFeelGood is therefore "rows the feel-good filter dropped *after* opinion passed on them this run," not "all feel-good content seen." Applies stamped/residue/mixed paths equally. See plan Operational Notes for the operator-runbook version. Per-attempt [digest] brief filter drops log is intentionally unchanged — it does not carry dropped_opinion= today and must not gain dropped_feelgood= either (C1). 14 source-textual tests in greenfield tests/digest-buildDigest-feelgood-filter.test.mjs + no regressions across 218 tests in brief-from-digest-stories / brief-llm / seed-envelope-parity sweep. * chore(deploy): COPY feelgood-classifier into digest-notifications image (U4) Without this COPY the Railway cron crashes at startup with ERR_MODULE_NOT_FOUND on the new '../server/_shared/feelgood-classifier.js' import that U3 added to scripts/seed-digest-notifications.mjs. Same failure mode the transitive-import-closure test caught for the opinion-classifier in PR #3690 — that test still passes here by construction. * test(carousel): bump PNG-render per-test timeout (orthogonal to feel-good filter) PNG cold-render (font + resvg-wasm init) measures ~10s on a warm machine and longer under concurrent test load. The node:test default per-test timeout (5s in tsx --test) false-fails these three tests in the pre-push full-suite sweep while they pass in isolation. Bumping to 30s removes the false failure without masking real regressions. Pre-existing flake, unrelated to this PR's feel-good filter — surfaced because adding new tests/*.test.mjs files flipped the pre-push hook into RUN_ALL mode. * fix(brief): bump rss:feed cache prefix v3→v4 to close pre-PR ParseResult leakage PR-review finding: pre-PR cached ParseResults in rss:feed:v3 contain ParsedItems without isFeelGood. If a cache hit returned one during the 1h healthy-cache rollout window, buildStoryTrackHsetFields would write `'isFeelGood', undefined ? '1' : '0'` → '0' onto the fresh story:track:v1 row. buildDigest's stampMissing check (`typeof !== 'string' || length === 0`) treats '0' as a genuine "not feel-good" verdict and skips the residue catch — feel-good content could silently slip through for up to one hour post-deploy. Fix: bump the cache prefix v3→v4 so every pre-PR ParseResult is invalidated on deploy. The first post-deploy cron tick is a cold read for every feed; fresh parseRssXml runs stamp isFeelGood correctly before buildStoryTrackHsetFields ever sees the item. Mirrors the v2→v3 precedent on this same cache for the same class of parsed-cache-shape change. Same latent bug exists in PR #3690's isOpinion path (1h leakage window after that PR shipped, masked by the TTL). A separate backport can address it; out of scope for this PR. New test: tests/news-story-track-description-persistence.test.mts asserts the v4 cache prefix is present in fetchAndParseRss source — locks the cutover so a refactor cannot silently revert to v3. Updated comment on the existing missing-field test to clarify that the '0' fallback is buildStoryTrackHsetFields' own behavior; the cache-leakage failure mode is closed by the v4 prefix bump above.
Summary
PR C of the brief-pipeline plan (
docs/plans/2026-05-14-001-…, Phase 3 F3 + Phase 4 F4). Stacked on PR B (#3688) → basefix/brief-grounding-gate-adapter; auto-retargets up the chain as A→B merge.F3 — opinion/analysis exclusion
The May 14 brief ranked a Le Monde opinion column ("'Russia's invasion of Ukraine could have warned Trump…'", by columnist Gilles Paris) as story #1, tagged Critical, ahead of a nuclear ICBM test. The brief is event-driven intelligence — an op-ed column is not an event.
server/_shared/opinion-classifier.js— single shared classifierclassifyOpinion({ title, link, description }). Tiered: STRONG signals (URL/opinion/,/views/,/commentary/,/editorial/,/op-ed/,/columnists/segments; explicitOpinion:/Analysis:/Commentary:/Op-Ed:headline prefix) classify alone; CORROBORATING signals (whole-headline quote-wrap, description framing words,/analysis/URL) need a STRONG signal OR two corroborating. Conservative — a false negative ships one op-ed; a false positive silently drops a real event.story:track:v1row carry neither — onlytitle,link,description. Both layers classify from the same three signals.list-feed-digest.ts) —ParsedItemgainsisOpinion, set byclassifyOpinioninparseRssXml;buildStoryTrackHsetFieldsstampsisOpinion('1'/'0') on thestory:track:v1row.buildDigest) — dropsisOpinion === '1'rows; for residue rows ingested before the stamp shipped (noisOpinionfield), re-classifies from the persisted title/link/description. Emits[digest] buildDigest opinion filter dropped Ntelemetry.story:trackrows feed more than the brief; onlybuildDigest(the brief's read path) filters.F4 — lead ↔ final-card-#1 coherence
The synthesis emits
leadandrankedStoryHashesindependently, with no constraint that the lead is about the rendered first story. AndrankedStoryHashes[0]≠data.stories[0]—orderBriefCandidatesre-sorts by severity/topic/score after the LLM rank. On May 14 the lead was about the Ukraine-energy story whiledata.stories[0]was the Le Monde column.leadGroundsAgainstStory(lead, headline)inbrief-llm.mjs— reuses thecheckLeadGroundinganchor machinery but with a fixed threshold of 1.checkLeadGroundingis the wrong fit: scoped to one story, a single headline can carry ≥4 anchors, tripping its size-based threshold up to 2 — too strict for a coherence question ("same story?", not "how grounded?").composeAndStoreBriefForUserruns the check aftercomposeBriefFromDigestStories, againstenvelope.data.stories[0](the final ordered first card) — neverrankedStoryHashes[0]. It runs in the orchestration layer, not the pure composer, socomposeBriefFromDigestStoriesstays I/O-free. Measure-first (plan F4 option b): emits[digest] lead card1 coherenceevery brief + aLEAD/CARD-#1 INCOHERENCEwarn on mismatch; ships the brief as-is. Once the mismatch rate is known, decide between regenerating the lead bound tostories[0]or a separateleadStoryHash.Test coverage
classifyOpinionunit tests — STRONG-alone, two-corroborating, one-corroborating-not-enough, the verbatim May 14 Le Monde column → opinion,/analysis/hard-news NOT dropped, quoted-phrase (not whole-wrap) hard-news NOT dropped, bare-noun "Opinion polls…" NOT caught, input safetyparseRssXmlcarriesisOpinion(hard-news false / op-ed true)buildStoryTrackHsetFieldsstampsisOpinion'1'/'0'+ legacy missing-field →'0'leadGroundsAgainstStorytests incl. the May 14 F4 regression (lead about Netanyahu vs Le Monde card-Batch market and RSS fetching with progressive updates #1 → incoherent) and degenerate no-anchor → skipDockerfile.digest-notificationsCOPYs the newopinion-classifier.js— the transitive-import-closure test caught the missing COPYTest plan
node --test tests/opinion-classifier.test.mjs→ 11/11node --test tests/brief-llm.test.mjs→ 100/100tsx --test tests/news-rss-description-extract.test.mts tests/news-story-track-description-persistence.test.mts→ 26/26tsx --test tests/*.test.mjs tests/*.test.mts→ 8728/8728npx tsc --noEmit+tsc --noEmit -p tsconfig.api.jsoncleanbiome check— 3 pre-existing warnings (buildDigest/main complexity already 5x over limit; one useOptionalChain on untouched list-feed-digest code), none introduced herebuildDigest opinion filter dropped N; no op-ed/column in the rendered brieflead card1 coherencelines; measure thecoherent=falserate to decide F4 follow-upNotes
No cache-prefix bump — F3 changes digest pool membership (fewer stories), not the synthesis cache material; F4 is pure telemetry.
Depends on PR A (#3685) and PR B (#3688). This is the last code PR in the plan; PR D (date drift) and PR E (threads consistency) remain as deferred low-urgency follow-ups.