fix(insights): trust cluster rank, stop LLM from re-picking top story#3358
Conversation
WORLD BRIEF panel published "Iran's new supreme leader was seriously wounded, leading him to delegate power to the Revolutionary Guards. This development comes amid an ongoing war with Israel." to every visitor for 3h. Payload: openrouter / gemini-2.5-flash. Root cause: callLLM sent all 10 clustered headlines with "pick the ONE most significant and summarize ONLY that story". Clustering ranked Lebanon journalist killing #1 (2 corroborating sources); News24 Iran rumor ranked #3 (1 source). Gemini overrode the rank, picked #3, and embellished with war framing from story #4. Objective rank (sourceCount, velocity, isAlert) lost to model vibe. Shrink the LLM's job to phrasing. Clustering already ranks — pass only topStories[0].primaryTitle and instruct the model to rewrite it using ONLY facts from the headline. No name/place/context invention. Also: - temperature 0.3 -> 0.1 (factual summary, not creative) - CACHE_TTL 3h -> 30m so a bad brief ages out in one cron cycle - Drop dead MAX_HEADLINES const Payload shape unchanged; frontend untouched.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR fixes a real confabulation bug by narrowing the LLM's job from "rank 10 headlines and pick one" to "rewrite the already-ranked top headline," preventing small models from overriding the objective clustering score. The prompt, temperature, and One tradeoff worth tracking: the Confidence Score: 5/5Safe to merge — the fix is well-targeted, payload shape is unchanged, and the only finding is a P2 tradeoff the author is aware of. No P0 or P1 issues found. The topStories[0] access is guarded, the LLM fallback chain is intact, and LKG preservation still works for the degraded path. The single P2 note (CACHE_TTL regression) is a conscious engineering tradeoff documented in the PR description. scripts/seed-insights.mjs — CACHE_TTL change worth re-evaluating before merge Important Files Changed
Sequence DiagramsequenceDiagram
participant Cron
participant fetchInsights
participant clusterItems
participant selectTopStories
participant callLLM
participant Redis
Cron->>fetchInsights: run seed
fetchInsights->>Redis: GET news:digest:v1:full:en
Redis-->>fetchInsights: digest items
fetchInsights->>clusterItems: normalizedItems[]
clusterItems-->>fetchInsights: clusters[] (ranked by sourceCount + velocity + isAlert)
fetchInsights->>selectTopStories: clusters, 8
selectTopStories-->>fetchInsights: topStories[] (pre-ranked)
Note over fetchInsights: topHeadline = topStories[0].primaryTitle (trust cluster rank)
fetchInsights->>callLLM: topHeadline (single string)
Note over callLLM: OLD: array of 10, LLM picks + summarizes
Note over callLLM: NEW: single headline, LLM only rewrites
callLLM-->>fetchInsights: { text, model, provider }
fetchInsights->>Redis: SET news:insights:v1 (TTL 1800s)
Reviews (1): Last reviewed commit: "fix(insights): trust cluster rank, stop ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| const CACHE_TTL = 10800; // 3h — 6x the 30 min cron interval (was 1x = key expired on any missed run) | ||
| const MAX_HEADLINES = 10; | ||
| const CACHE_TTL = 1800; // 30m — matches cron interval; bad briefs age out in one cycle instead of persisting 3h |
There was a problem hiding this comment.
The previous CACHE_TTL of 3h was explicitly set to be 6× the cron interval with the comment "was 1x = key expired on any missed run." Resetting to 30m (1×) means a single failed cron execution (transient LLM timeout, Redis hiccup, digest not ready) leaves users with no worldBrief until the next cycle completes successfully. The LKG preservation at line 336 only activates when status === 'degraded' (no LLM available), not when the run throws before reaching that point. Consider a slightly longer TTL (e.g. 2700s = 1.5× cron) to stay resilient while still evicting a bad brief within two cycles.
…HERE Follow-up to review feedback on the ranking contract, TTL, and prompt: 1. Corroboration gate (P1a). scoreImportance() in scripts/_clustering.mjs is keyword-heavy (violence +125 on a single word, flashpoint +75, ^1.5 multiplier when both hit), so a single-source sensational rumor can outrank a 2-source lead purely on lexical signals. Blindly trusting topStories[0] would let the ranker's keyword bias still pick bad stories. Walk topStories for sourceCount >= 2 instead — corroboration becomes a hard requirement, not a tiebreaker. If no cluster qualifies, publish status=degraded with no brief (frontend already handles this). 2. CACHE_TTL back to 10800 (P1b). 30m TTL == one cron cadence means the key expires on any missed or delayed run and /api/bootstrap loses insights entirely (api/bootstrap.js reads news:insights:v1 directly, no LKG across TTL-gap). The short TTL was defense-in-depth for bad content; the real safety is now upstream (corroboration gate + grounded prompt), so the LKG window doesn't need to be sacrificed for it. 3. Prompt: location conditional (P2). "Use ONLY facts present" + "Lead with WHAT happened and WHERE" conflicted for headlines without an explicit location and pushed the model toward inferred-place hallucination. Replaced with "Include a location, person, or organization ONLY if it appears in the headline."
Review P2: the corroboration gate and the prompt's no-invention rules had no tests, so future edits to selectTopStories() ordering or prompt text could silently reintroduce the original hallucination. Extract the brief-selection helper and prompt builders into a pure module (scripts/_insights-brief.mjs) so tests can import them without triggering seed-insights.mjs's top-level runSeed() call: - pickBriefCluster(topStories) returns first sourceCount>=2 cluster - briefSystemPrompt(dateISO) returns the system prompt - briefUserPrompt(headline) returns the user prompt Regression tests (tests/seed-insights-brief.test.mjs, 12 cases) lock: - pickBriefCluster skips single-source rumors even when ranked above a multi-sourced lead (explicit regression: News24 Iran supreme leader 2026-04-23 scenario with realistic scores) - pickBriefCluster tolerates missing/null entries - briefSystemPrompt forbids invented facts and proper nouns - briefSystemPrompt's "location" rule is conditional (no unconditional "Lead with WHAT and WHERE" directive that would push the model toward place-inference when the headline has no location) - briefSystemPrompt does not contain "pick the most important" style language (ranking is done by pickBriefCluster upstream) - briefUserPrompt passes the headline verbatim and instructs "only facts from this headline" Also fix a misleading comment on CACHE_TTL: corroboration is gated at brief-selection time, not on the topStories payload itself (which still includes single-source clusters rendered as the headline list). test:data: 6657/6657 pass (was 6645; +12).
1 participant
Summary
Two compounding issues produced a user-visible hallucination in the WORLD BRIEF panel. This PR fixes both, with a corroboration gate as the safety backstop — and locks both behaviors with regression tests.
1. The LLM was double-ranking.
scripts/seed-insights.mjssent all 10 clustered headlines to gemini-2.5-flash with "pick the ONE most significant and summarize." Shrink that to phrasing-only — pass a single chosen headline and instruct the model to rewrite using only facts present in it.2. The upstream cluster ranker is keyword-heavy, not objective.
scoreImportance()inscripts/_clustering.mjsawardssourceCount * 10but+100+25per violence keyword,+60+75for flashpoint/military, and ax1.5multiplier when both hit — a single-source sensational rumor routinely outranks a 2-source lead. So blindly trustingtopStories[0]still produces bad briefs. Add a corroboration gate: walktopStoriesfor the first cluster withsourceCount >= 2; if none qualifies, publishstatus: 'degraded'with no brief. This gate only affects the brief paragraph — the topStories list itself continues to render single-source clusters.3. Prompt contradiction. "Use ONLY facts present in the headline" conflicted with "Lead with WHAT happened and WHERE" for headlines with no explicit location — pushing the model toward inferred-place hallucination. Replaced with "Include a location, person, or organization ONLY if it appears in the headline."
4. Regression tests. Selection logic + prompt invariants extracted into
scripts/_insights-brief.mjs(pure module, no side effects at import time) and locked bytests/seed-insights-brief.test.mjs— 12 cases including an explicit News24-Iran-scenario test that fails if future code ever briefs a single-source cluster over a multi-sourced lead.Also:
temperature 0.3 -> 0.1(factual summary), deadMAX_HEADLINESconst removed. CACHE_TTL unchanged at 10800 (3h) — a one-cron-cadence TTL would reintroduce the cache-drop outage the old comment warned against (/api/bootstrapreads the key directly). Bad content is gated at brief-selection time now, so the LKG window does not need to be sacrificed.Payload shape unchanged; frontend untouched.
Why (user-visible bug)
WORLD BRIEF panel published for ~3h on 2026-04-23:
Payload receipt:
briefProvider: openrouter,briefModel: google/gemini-2.5-flash. An earlier generation also confabulated the name "Ayatollah Ali Khamenei" (not in any input headline — gemini filled it in from its training prior).Input top stories (cluster rank):
Old prompt told gemini to pick the most significant and summarize it. Gemini picked #3 (single-source rumor) over #1 (multi-sourced lead) and embellished with the "ongoing war with Israel" frame pulled from #4. After this PR: #3 is never eligible for the brief (sourceCount < 2); #1 is briefed using only its own headline text.
Test plan
npm run typecheck— PASS (22s)npm run typecheck:api— PASS (8s)npm run lint— PASS (no new warnings beyond 202-warning / 23-info baseline)npm run test:data— PASS (6657/6657 — +12 new regression tests)node --test tests/seed-insights-brief.test.mjs— 12/12 PASS including the News24-Iran regression scenarionode --test tests/edge-functions.test.mjs— PASSapi/*.js— PASSnpm run lint:md— PASS (0 errors)npm run version:check— PASSscripts/*.cjs— PASSnode --check scripts/seed-insights.mjs+node --check scripts/_insights-brief.mjsDEL news:insights:v1in Upstash so the next cron tick regenerates immediately (current bad brief otherwise persists until its existing TTL expires)/api/bootstrappayload:insights.worldBriefsummarizes asourceCount >= 2cluster and contains no proper noun absent from that cluster'sprimaryTitle; orinsights.status === 'degraded'with emptyworldBrief