fix(classifier): two-layer historical-retrospective downgrade (L1 + L3)#3429
Conversation
Brief 2026-04-26-1302 surfaced a 40-year Chernobyl anniversary article from
Live Science:
"Science history: Chernobyl nuclear power plant melts down, bringing the
world to the brink of disaster — April 26, 1986 - Live Science"
Investigation showed two distinct failure modes that cooperate to ship
retrospective/anniversary content as if it were a current crisis:
L1 — enrichWithAiCache skipped the LLM cache result for keyword=critical
items (`if (0.9 <= item.confidence) continue;`). Original intent was
"trust keyword when confident", but for retrospective titles that
trip a CRITICAL keyword (e.g. "meltdown"), the LLM is the only thing
that could disambiguate context — and the gate locked it out. Removed.
The U4 +2-tier cap already protects against UPWARD over-promotion;
symmetric DOWNWARD demotion is unconditionally safer than keeping a
suspect keyword classification (capLlmUpgrade is a Math.min so
downgrades pass freely). Cache application also now covers the new
'keyword-historical-downgrade' classSource so LLM can confirm/override
the heuristic.
L3 — Two-layer historical-marker downgrade:
L3a (classifier-side): classifyByKeyword now downgrades CRITICAL/HIGH
keyword matches to info when the title contains a retrospective
marker. Markers (cheap regex):
- Prefix: ^(Science history|On this day|Today in|This day in|
Throwback|Flashback)
- Phrase: \b(N years/decades/months ago|after|later|anniversary|
in memoriam|remembering|commemorat|retrospective)\b
- Full date: "April 26, 1986" or ISO 1986-04-26
Standalone 4-digit year is intentionally NOT a marker (current-event
headlines like "Russia warns of 2026 strike" would falsely trigger).
Downgrade applies only to CRITICAL/HIGH; LOW/MEDIUM are left alone
(don't clear brief thresholds anyway, over-aggression cost outweighs
signal). Distinct source tag 'keyword-historical-downgrade' for
telemetry.
L3b (LLM-cache-side defense-in-depth): The Chernobyl headline ABOVE
doesn't match any CRITICAL keyword in the keyword classifier
("meltdown" substring isn't present in "melts down" — there's a
space). So L3a wouldn't catch it. But the LLM cache had promoted
this title (or one with the same hash) to CRITICAL at some prior
point, and enrichWithAiCache applied that cache hit. The new
L3b guard re-runs hasHistoricalMarker AFTER capLlmUpgrade — if the
LLM-promoted level is CRITICAL/HIGH AND the title has a marker,
force info. Logs on every fire so operators can audit.
Tests:
- tests/news-classifier-historical-downgrade.test.mts — 23 cases for
hasHistoricalMarker predicate matrix + classifyByKeyword integration.
Includes regression guards: current-year mentions don't trigger,
"history" alone doesn't trigger (only the prefix forms), LOW/MEDIUM
not downgraded, current-event critical headlines unchanged.
- tests/news-classifier-llm-historical-guard.test.mts — 9 cases
covering the LLM-cache guard predicate behavior + behavioral
semantics doc (CRITICAL+marker → info; HIGH+marker → info;
MEDIUM+marker → unchanged; CRITICAL-no-marker → unchanged). Includes
the EXACT brief 2026-04-26-1302 title as a test fixture.
Type tightening:
- ClassificationResult.source: 'keyword' | 'keyword-historical-downgrade'
- ParsedItem.classSource: 'keyword' | 'keyword-historical-downgrade' | 'llm'
106/106 tests pass across the touched + parity-coupled surface;
importance-score-parity preserved.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds a two-layer defense against historical/retrospective articles shipping as critical news:
Confidence Score: 3/5Not safe to merge without addressing the unconditional confidence-gate removal, which creates a new suppression vector for genuine critical events via stale LLM cache entries. One P1 finding (stale-LLM demotion of genuine critical events, introduced by blanket gate removal) pulls the score below the P1 ceiling of 4. The targeted fix is small, but the risk is on the highest-severity classification path. server/worldmonitor/news/v1/list-feed-digest.ts — specifically the removed confidence skip and the interaction between capLlmUpgrade and downgrades for non-retrospective items. Important Files Changed
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| for (const item of relatedItems) { | ||
| if (0.9 <= item.confidence) continue; | ||
| // L1 (PR #3424): the prior `if (0.9 <= item.confidence) continue` skip | ||
| // here meant the LLM cache result was IGNORED for keyword=critical | ||
| // matches. That made the cache an upgrade-only path: keyword=info → | ||
| // LLM=high could promote, but keyword=critical → LLM=info (e.g. | ||
| // retrospective/anniversary content tripping a critical keyword) | ||
| // could NEVER demote. Removed unconditionally — the U4 +2 tier cap | ||
| // below already protects against poisoned UPWARD promotions, and | ||
| // symmetric DOWNGRADES are unconditionally safer than keeping a | ||
| // suspect keyword classification. | ||
| // | ||
| // Cap the LLM upgrade at +2 tiers above the keyword classification | ||
| // so a poisoned cache entry (e.g., "About Section 508" → high) can't | ||
| // promote an info-keyword item past medium (info+2=medium). Legitimate | ||
| // medium→critical upgrades (medium+2=critical) remain reachable; the | ||
| // bounded loss is keyword=low → LLM=critical, which caps at high | ||
| // (low+2=high) and is logged below. See LEVEL_RANK doc + R4 for the | ||
| // full per-keyword cap table. | ||
| const cappedLevel = capLlmUpgrade(item.level, hit.level); | ||
| // medium→critical upgrades (medium+2=critical) remain reachable. | ||
| // capLlmUpgrade is a Math.min so downgrades pass through freely. | ||
| // See LEVEL_RANK doc + R4 for the full per-keyword cap table. | ||
| let cappedLevel = capLlmUpgrade(item.level, hit.level); |
There was a problem hiding this comment.
The prior confidence >= 0.9 skip specifically protected high-confidence keyword matches from being demoted by the LLM cache. Its removal is unconditional here: capLlmUpgrade only caps upward promotions (Math.min semantics); downgrades pass through freely. The L3b guard added below only saves items whose title contains a historical marker, so a stale or incorrectly-populated cache entry for a genuine current breaking event will demote it to info with no remaining safeguard.
The targeted fix needed to address retrospectives is narrower: bypass the gate only when an item was already classified as keyword-historical-downgrade, since those are the items that need LLM confirmation. Items classified as a plain critical keyword match would retain the skip, preventing stale demotions of genuine events. Removing the gate unconditionally is riskier than the original bug it addresses.
There was a problem hiding this comment.
Addressed in commit 6024206 — restored if (0.9 <= item.confidence) continue; after the L3 marker check. The retrospective case PR #3424 originally targeted is handled upstream by classSource='keyword-historical-downgrade' (confidence 0.85, level=info), which still flows through enrichWithAiCache and is caught by the L3 marker guard. Items reaching with confidence 0.9 are by construction keyword=critical with no marker — those keep their keyword verdict. Brief 2026-04-26-1302 "Science history: melts down…" still works because keyword=info no-match (confidence 0.3) bypasses the gate and L3 forces info.
| const FULL_DATE_RE = | ||
| /\b(?:january|february|march|april|may|june|july|august|september|october|november|december)\s+\d{1,2},?\s+(?:19|20)\d{2}\b/i; | ||
|
|
||
| const ISO_DATE_RE = /\b(?:19|20)\d{2}-\d{1,2}-\d{1,2}\b/; |
There was a problem hiding this comment.
ISO_DATE_RE (and FULL_DATE_RE) match current-year dates, risking false-positive downgrades
ISO_DATE_RE uses (?:19|20)\d{2} which covers 2000–2099, including the current year. A current-event headline containing today's date in ISO format would match and incorrectly downgrade a CRITICAL/HIGH keyword match to info. The same applies to FULL_DATE_RE. The test suite validates that a year alone is safe but has no negative test for a current-year full date.
One mitigation: restrict the year range in both regexes to a cutoff well before the current year, accepting that very recent anniversaries might be missed — these are rare, while false positives on current-event headlines are an ongoing suppression risk.
There was a problem hiding this comment.
Already addressed in round 2 (commit 96d3c12). Both FULL_DATE_RE and ISO_DATE_RE matches are gated through isPastRetrospectiveYear(year, nowMs) (_classifier.ts:270), which requires year < currentYear - 1. So a current-year ISO date in a headline (e.g. 2026-04-26) does NOT trigger the marker. There's also a regression test for the FULL_DATE current-year case in tests/news-classifier-llm-historical-guard.test.mts; I can add a parallel ISO-format negative test if you want belt-and-suspenders.
| /** | ||
| * Returns true if the title looks like a historical retrospective. | ||
| * Used by classifyByKeyword to downgrade CRITICAL/HIGH keyword matches | ||
| * (e.g. "meltdown") that appear in a backward-looking headline. | ||
| * | ||
| * Exported for test coverage — DO NOT call from production code paths | ||
| * other than classifyByKeyword. | ||
| */ |
There was a problem hiding this comment.
The JSDoc says "DO NOT call from production code paths other than classifyByKeyword", but hasHistoricalMarker is now also directly imported and called from enrichWithAiCache in list-feed-digest.ts. This comment will mislead future readers about the intended call sites and should be updated to reflect both approved callers.
There was a problem hiding this comment.
The JSDoc on commit 6024206 reads DO NOT call from production code paths other than classifyByKeyword and enrichWithAiCache — enrichWithAiCache is already named. Looks like a stale diff context on Greptile's side; the JSDoc was updated when the second caller was added.
Reviewer found two false-positive vectors that suppressed real critical
alerts:
- "Today in Ukraine: Russian missile strikes Kyiv" → was downgraded
to info because of bare "Today in" prefix. This is a current-event
headline pattern, not a historical one.
- "Missile launch reported on April 26, 2026" → was downgraded
because any full-date pattern matched, regardless of year.
Both vectors would have suppressed real missile/attack/invasion alerts
before they reached the brief.
Fixes:
1. Prefix regex narrowed:
REMOVED: bare "Today in" / "This day in" (too broad — legitimate
current-event uses).
KEPT: "Science history:", "Throwback", "Flashback" (always
retrospective).
ADDED: "On this day in YYYY" (year required — prevents bare
"On this day, Iran fires missile" from triggering).
ADDED: "This day in history" (specific phrasing — distinct from
bare "This day in").
2. Full-date matching now extracts the year and only treats dates as
retrospective when year < currentYear - 1 (i.e. 2+ years old). In
2026:
- 2024 and earlier dates → retrospective marker fires.
- 2025 (last year) → NOT a marker (could be current context, e.g.
"court ruling on April 15, 2025 takes effect").
- 2026 (current) → NOT a marker.
- 2027+ (future) → NOT a marker (clock skew or scheduled events).
Same logic applies to ISO date format.
3. hasHistoricalMarker(title, nowMs?) now takes optional nowMs for
unit testability. Defaults to Date.now(); production callers omit.
Tests:
- 4 explicit reviewer-fix safety cases:
* "Today in Ukraine: Russian missile strikes Kyiv" → high (preserved)
* "Missile launch reported on April 26, 2026" → high (preserved)
* "Today in tech: Apple unveils iPhone" → no marker
* "On this day, Iran invasion begins" (no year) → no marker
- 4 full-date boundary cases (1986, 2024, 2025, 2026, 2027) verify
the year-based gating.
- All test calls pass NOW = 2026-04-15 UTC to pin "current year"
deterministically.
120/120 tests pass; importance-score-parity preserved.
… PR #3429 round 3) Previously the L3 defense-in-depth marker check ran on cappedLevel and only fired for critical/high. For keyword=info + hit=critical, capLlm- Upgrade demotes to medium (info+2=medium) — the post-cap check missed it and the brief 2026-04-26-1302 Chernobyl case shipped at MEDIUM, which still ships in 'all'-sensitivity briefs. Move the check to run on the RAW hit before the cap and force info unconditionally (not just critical/high). Retrospective markers now suppress the LLM verdict at every non-info level.
…1 PR #3429 round 4) Greptile flagged that removing the confidence>=0.9 skip unconditionally opens genuine current keyword=critical events to silent demotion: the LLM cache hit feeds capLlmUpgrade which is Math.min, so a stale/wrong cache entry saying 'info' demotes critical -> info with no safeguard. The retrospective case PR #3424 wanted to handle is already handled UPSTREAM in classifyByKeyword via classSource='keyword-historical- downgrade' (confidence 0.85, level=info), which still flows through this function and is caught by the L3 marker check. Items reaching enrichWithAiCache at confidence 0.9 are by construction keyword=critical matches where the keyword classifier saw NO marker - trust the keyword verdict for those. The L3 marker check still runs BEFORE this skip, so the keyword=info (no-match, confidence 0.3) + marker case - the brief 2026-04-26-1302 'Science history: melts down...' shape - still gets forced to info.
…ix (#3480) * fix(classifier): match flashback/throwback after publisher brand prefix Brief 2026-04-28-0801 surfaced "CBS News Radio flashback: D-Day, Invasion of Normandy in 1944" as severity=critical in slot #3, two days after PR #3429 shipped the historical-retrospective downgrade. Root cause: HISTORICAL_PREFIX_RE was anchored with `^` — `/^(?:science history|throwback|flashback)\s*:?/i` — so flashback / throwback only matched at title position 0. Publishers prepend their brand ("CBS News Radio", "BBC", "NPR") before the marker token, which defeated the anchor. The keyword path then matched `invasion` unimpeded → critical. Fix: word-boundary match for flashback / throwback (unambiguous markers regardless of position); science history stays anchored since it's an editorial-slot prefix that's only retrospective at the start of a headline. /(?:^science history\s*:?|\b(?:throwback|flashback)\b)/i Regression test added under the existing news-classifier-historical-downgrade suite covering CBS News Radio, BBC, and NPR brand-prefix forms. All 56 cases in that file plus the 56 cases across the historical-downgrade + LLM-cache-guard tests still pass. * fix(classifier): tighten flashback/throwback to editorial-slot syntax Reviewer P2 on PR #3480: widening flashback/throwback to bare word-boundary match also broadens the L3b LLM-cache guard at list-feed-digest.ts:547, which force-demotes ANY cached LLM hit to info when hasHistoricalMarker is true. A current-event headline that incidentally uses "flashback" / "throwback" as a comparison word ("Markets see flashback to 2008 crisis as bonds tumble") would be wrongly suppressed even though the marker isn't acting as a retrospective framing token. Replace the broad word-boundary form with two explicit branches: 1. ANCHORED: marker at title position 0 (original behavior — "Throwback Thursday:", "Flashback:", "Science history:"). 2. BRAND-PREFIX: 1-4 Title-Case prefix words + marker + optional qualifier word + colon. Catches the brand-prefixed forms ("CBS News Radio flashback:", "BBC Throwback Thursday:", "NPR Flashback Friday:") while rejecting sentence-form uses. The brand-prefix branch deliberately requires (a) Title-Case prefix words (rejects "markets see flashback…"), AND (b) an editorial-slot colon after the marker (rejects "Markets See Flashback To 2008 Crisis As Bonds Tumble"). The Title-Case gate uses a non-i regex so [A-Z] enforces actual capitalization rather than getting case-folded. Adds 5 negative tests for the realistic false-positive cases the reviewer was worried about — "Markets see flashback to 2008 crisis", "Stocks suffer flashback to March 2020 crash", "Tesla stock throwback after split", "AI flashback to 2023 boom: Nvidia earnings beat", and a lowercase-leading sentence form. All existing positive tests still pass (61/61 in the historical- downgrade + LLM-cache-guard suite). * chore(classifier): bump CLASSIFY_CACHE_PREFIX v4 → v5 Evicts cache entries that landed under the pre-publisher-prefix-fix classifier — brand-prefixed retrospective titles like "CBS News Radio flashback: D-Day, Invasion of Normandy in 1944" had been promoted to severity=critical via the `invasion` keyword and persisted in the classify cache. The new brand-prefix branch in _classifier.ts (PR #3480 commits 5ec9892 + f61fae5) re-rules those rows on next touch; v5 forces immediate eviction rather than waiting for natural TTL on every poisoned entry. Three lockstep sites updated atomically per the cache-prefix-bump- propagation-scope learning: - server/worldmonitor/intelligence/v1/_shared.ts:20 (canonical writer) - server/worldmonitor/news/v1/list-feed-digest.ts:511 (doc-comment) - scripts/ais-relay.cjs:3172 (relay inline helper) The news-classify-cache-prefix-audit static-analysis test enforces lockstep — passes after this change.
…ix (koala73#3480) * fix(classifier): match flashback/throwback after publisher brand prefix Brief 2026-04-28-0801 surfaced "CBS News Radio flashback: D-Day, Invasion of Normandy in 1944" as severity=critical in slot #3, two days after PR koala73#3429 shipped the historical-retrospective downgrade. Root cause: HISTORICAL_PREFIX_RE was anchored with `^` — `/^(?:science history|throwback|flashback)\s*:?/i` — so flashback / throwback only matched at title position 0. Publishers prepend their brand ("CBS News Radio", "BBC", "NPR") before the marker token, which defeated the anchor. The keyword path then matched `invasion` unimpeded → critical. Fix: word-boundary match for flashback / throwback (unambiguous markers regardless of position); science history stays anchored since it's an editorial-slot prefix that's only retrospective at the start of a headline. /(?:^science history\s*:?|\b(?:throwback|flashback)\b)/i Regression test added under the existing news-classifier-historical-downgrade suite covering CBS News Radio, BBC, and NPR brand-prefix forms. All 56 cases in that file plus the 56 cases across the historical-downgrade + LLM-cache-guard tests still pass. * fix(classifier): tighten flashback/throwback to editorial-slot syntax Reviewer P2 on PR koala73#3480: widening flashback/throwback to bare word-boundary match also broadens the L3b LLM-cache guard at list-feed-digest.ts:547, which force-demotes ANY cached LLM hit to info when hasHistoricalMarker is true. A current-event headline that incidentally uses "flashback" / "throwback" as a comparison word ("Markets see flashback to 2008 crisis as bonds tumble") would be wrongly suppressed even though the marker isn't acting as a retrospective framing token. Replace the broad word-boundary form with two explicit branches: 1. ANCHORED: marker at title position 0 (original behavior — "Throwback Thursday:", "Flashback:", "Science history:"). 2. BRAND-PREFIX: 1-4 Title-Case prefix words + marker + optional qualifier word + colon. Catches the brand-prefixed forms ("CBS News Radio flashback:", "BBC Throwback Thursday:", "NPR Flashback Friday:") while rejecting sentence-form uses. The brand-prefix branch deliberately requires (a) Title-Case prefix words (rejects "markets see flashback…"), AND (b) an editorial-slot colon after the marker (rejects "Markets See Flashback To 2008 Crisis As Bonds Tumble"). The Title-Case gate uses a non-i regex so [A-Z] enforces actual capitalization rather than getting case-folded. Adds 5 negative tests for the realistic false-positive cases the reviewer was worried about — "Markets see flashback to 2008 crisis", "Stocks suffer flashback to March 2020 crash", "Tesla stock throwback after split", "AI flashback to 2023 boom: Nvidia earnings beat", and a lowercase-leading sentence form. All existing positive tests still pass (61/61 in the historical- downgrade + LLM-cache-guard suite). * chore(classifier): bump CLASSIFY_CACHE_PREFIX v4 → v5 Evicts cache entries that landed under the pre-publisher-prefix-fix classifier — brand-prefixed retrospective titles like "CBS News Radio flashback: D-Day, Invasion of Normandy in 1944" had been promoted to severity=critical via the `invasion` keyword and persisted in the classify cache. The new brand-prefix branch in _classifier.ts (PR koala73#3480 commits 5ec9892 + f61fae5) re-rules those rows on next touch; v5 forces immediate eviction rather than waiting for natural TTL on every poisoned entry. Three lockstep sites updated atomically per the cache-prefix-bump- propagation-scope learning: - server/worldmonitor/intelligence/v1/_shared.ts:20 (canonical writer) - server/worldmonitor/news/v1/list-feed-digest.ts:511 (doc-comment) - scripts/ais-relay.cjs:3172 (relay inline helper) The news-classify-cache-prefix-audit static-analysis test enforces lockstep — passes after this change.
1 participant
Why
Brief 2026-04-26-1302 surfaced this 40-year Chernobyl anniversary article from Live Science as if it were current news:
Investigation showed two distinct failure modes that cooperate to ship retrospectives as critical news:
L1: enrichWithAiCache locked out the LLM cache for keyword=critical items
Original intent: "trust keyword when confident." But for retrospective titles tripping a CRITICAL keyword (e.g.
meltdown), the LLM is the only thing that could disambiguate context — and the gate locked it out. The U4 +2-tier cap already protects against UPWARD over-promotion; symmetric DOWNWARD demotion is unconditionally safer than keeping a suspect keyword classification (capLlmUpgradeisMath.minso downgrades pass freely). Removed.L3: Two-layer historical-marker downgrade
L3a (classifier-side):
classifyByKeywordnow downgrades CRITICAL/HIGH keyword matches toinfowhen the title contains a retrospective marker. Cheap regex patterns:^(Science history|On this day|Today in|This day in|Throwback|Flashback)\b(N years/decades/months ago|after|later|anniversary|in memoriam|remembering|commemorat|retrospective)\bApril 26, 1986or ISO1986-04-26Standalone 4-digit year is intentionally NOT a marker (current-event headlines like "Russia warns of 2026 strike" would falsely trigger).
L3b (LLM-cache-side defense-in-depth): The Chernobyl headline above doesn't actually match any CRITICAL keyword in the keyword classifier —
"meltdown"substring isn't present in"melts down"(there's a space). So L3a wouldn't catch it. But the LLM cache had promoted this title at some prior point. The new L3b guard re-runshasHistoricalMarkerAFTERcapLlmUpgrade— if the LLM-promoted level is CRITICAL/HIGH AND the title has a marker, forceinfo. Logs on every fire so operators can audit.Tests
tests/news-classifier-historical-downgrade.test.mtshasHistoricalMarkerpredicate matrix +classifyByKeywordintegration. Regression guards: current-year mentions don't trigger,"history"alone doesn't trigger (only the prefix forms), LOW/MEDIUM not downgraded, current-event critical headlines unchangedtests/news-classifier-llm-historical-guard.test.mtsBehavioral matrix:
keyword-historical-downgradellmkeywordllmkeyword(unchanged — only CRITICAL/HIGH downgrade)106/106 tests pass; importance-score-parity preserved.
Type tightening
ClassificationResult.source:'keyword' | 'keyword-historical-downgrade'ParsedItem.classSource:'keyword' | 'keyword-historical-downgrade' | 'llm'Operator runbook
After merge, two new log lines surface guard fires:
[classify] LLM upgrade capped:(existing — U4 +2 cap)[classify] LLM-promoted level forced to info by historical marker:(new — L3b)Audit the second log over a 24h window post-deploy. If it fires on legitimate current-event titles (false positives), tighten the markers in a follow-up. Current marker set is intentionally conservative.
Test plan
track.titlematches a historical-marker pattern +track.severity ∈ {critical, high}Related