fix(brief-ingest): READ-time freshness + direct-RSS migration + U6 age mode#3422
Conversation
…e mode Three coupled gap closures discovered after PR #3419 merged. Brief 2026-04-26-0802 still surfaced two months-old Pentagon items DESPITE PR #3417's `when:1d` ingest gate working perfectly (verified live: the gated query returns zero Pentagon items in the last 24h). The cause was post-deploy residue, not the gate. See: skill: ingest-gate-tightening-leaves-residue-in-read-path Fixes (all on the same PR because they share the publishedAt persistence precondition): 1. READ-time freshness floor in buildDigest (seed-digest-notifications.mjs). Drops story:track:v1 rows whose source publishedAt is older than DIGEST_READ_MAX_AGE_HOURS (default 48h). Closes the residue window: when any future ingest-side gate is tightened, pre-deploy entries stop shipping in briefs even before they TTL-expire from the accumulator. Rows missing publishedAt (legacy pre-this-PR entries) fall through — back-compat. Operator kill switch: DIGEST_READ_MAX_AGE_HOURS=999. 2. Persist publishedAt in story:track:v1 HSET (list-feed-digest.ts). Adds 'publishedAt', String(item.publishedAt) to buildStoryTrackHsetFields. Required for #1 AND for U6's age-mode (#3) to actually find the field. Pre-existing rows pick it up on their next mention. Test: tests/news-story-track-description-persistence.test.mts asserts the field is emitted as a numeric string that round-trips through parseInt. 3. U6 audit gains --mode=age|url|both (audit-static-page-contamination.mjs). Original URL-pattern classifier is BLIND to Google-News-routed entries (track.link is news.google.com/rss/articles/CBMi opaque redirect). --mode=age matches by track.publishedAt > max-age-hours, the right primary signal for evicting post-deploy residue. Per-reason rollup surfaces url vs age hits. Operator runbook included in the script header. Default --mode=url is back-compat. 4. Feed swap: 3 of 5 gov sources to direct first-party RSS (_feeds.ts). Verified 2026-04-26 via live curl + end-to-end parseRssXml smoke test: - Pentagon -> https://www.war.gov/DesktopModules/ArticleCS/RSS.ashx?ContentType=1&Site=945 - White House (briefings) -> /briefings-statements/feed/ - White House Actions -> /presidential-actions/feed/ (NEW entry) Direct RSS means the publisher's pubDate is authoritative — no re-indexing of months-old PDFs creating fresh-looking Google entries. State Dept, Treasury, DOJ retain gn(...) with an explanatory comment; no working public RSS at any verified path, Federal Register fallback bot-blocked. The READ-time freshness floor (#1) mitigates the residue gap for those. 5. source-tiers.json: White House Actions = 1 (with scripts/shared mirror). New feed gets the same tier-1 source-boost as the existing White House entry; without it, Actions items would default to tier-4 with no boost. 250/250 tests pass; importance-score-parity preserved. Operator runbook for the immediate post-deploy residue: 1. Wait ~1h for cron tick to start writing publishedAt on existing entries via HSET re-mention. 2. Dry run: node scripts/audit-static-page-contamination.mjs --mode=age 3. Inspect output, confirm matched titles look stale. 4. Apply: --mode=age --apply
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR adds a READ-time
Confidence Score: 3/5Safe to merge for the structural fixes; the operator runbook misleads on residue cleanup scope, and the new White House Actions feed may silently produce no content One P1 (runbook --mode=age is a no-op for immediate pre-deploy residue, creating false cleanup confidence) and one P2 (White House Actions items exceed the ingest freshness gate at verification time). P1 ceiling is 4/5; the practical impact — stale items continuing to surface until TTL drains rather than being cleanly evicted — pulls the score to 3. scripts/audit-static-page-contamination.mjs (age-mode logic vs. legacy entries) and server/worldmonitor/news/v1/_feeds.ts (White House Actions freshness window) Important Files Changed
Sequence DiagramsequenceDiagram
participant RSS as Direct RSS / gn()
participant Ingest as list-feed-digest<br/>(ingest-time gate)
participant Redis as story:track:v1<br/>(Redis HSET)
participant Digest as seed-digest-notifications<br/>(READ-time gate)
participant User as User Brief
RSS->>Ingest: raw items (with pubDate)
Ingest->>Ingest: resolveMaxAgeMs() — drop if publishedAt > 48h
Ingest->>Redis: HSET publishedAt=String(item.publishedAt)
Redis->>Digest: HGETALL story:track:v1:*
Digest->>Digest: READ-time gate — drop if publishedAt present AND > 48h
note over Digest: legacy entries (no publishedAt) fall through
Digest->>User: filtered digest stories
note over Redis,Digest: Pre-deploy entries: no publishedAt → both READ-time gate and mode=age audit are blind
Reviews (1): Last reviewed commit: "fix(brief-ingest): READ-time freshness +..." | Re-trigger Greptile |
| return { byHost: fmt(hostCounts), byPathPattern: fmt(pathPatternCounts) }; | ||
| } | ||
|
|
||
| function classifyTrack(track, nowMs) { | ||
| const reasons = []; | ||
| const link = typeof track.link === 'string' ? track.link : ''; | ||
|
|
||
| if ((MODE === 'url' || MODE === 'both') && link && isInstitutionalStaticPage(link)) { | ||
| reasons.push('url'); | ||
| } | ||
|
|
||
| if (MODE === 'age' || MODE === 'both') { | ||
| const pubMs = Number.parseInt(track.publishedAt ?? '', 10); | ||
| if (Number.isInteger(pubMs) && pubMs > 0 && nowMs - pubMs > MAX_AGE_MS) { | ||
| reasons.push('age'); | ||
| } | ||
| } | ||
|
|
||
| return reasons; | ||
| } | ||
|
|
||
| async function main() { | ||
| const { url, token } = getRedisCredentials(); | ||
| console.log(`[audit] mode=${APPLY ? 'APPLY (will DELETE)' : 'DRY RUN'}`); | ||
| console.log( | ||
| `[audit] mode=${MODE} ${APPLY ? 'APPLY (will DELETE)' : 'DRY RUN'}` + | ||
| (MODE === 'age' || MODE === 'both' ? ` max-age-hours=${ARGS.maxAgeHours}` : ''), | ||
| ); | ||
| console.log(`[audit] scanning ${SCAN_PATTERN}…`); | ||
|
|
There was a problem hiding this comment.
--mode=age is a no-op for the immediate pre-deploy residue
The operator runbook (PR description, steps 3-4) directs operators to run --mode=age for the "immediate residue" — the months-old Pentagon/gov entries already in Redis. But those entries were written before this PR, which is the first commit that persists publishedAt to story:track:v1 HSET fields. They have no publishedAt key.
In classifyTrack, the age branch is:
const pubMs = Number.parseInt(track.publishedAt ?? '', 10);
if (Number.isInteger(pubMs) && pubMs > 0 && nowMs - pubMs > MAX_AGE_MS) {
reasons.push('age');
}Number.parseInt("", 10) → NaN; Number.isInteger(NaN) → false → no match. The same back-compat fallthrough that protects the buildDigest filter also makes the age-mode audit blind to every entry written before this deploy.
The runbook's step 4 — "matched titles should look stale (months-old)" — will see zero matches, giving false confidence that the residue has been evicted.
Pre-deploy stale Google-News-routed entries (the exact ones --mode=age was designed for) cannot be evicted by this tool until they are re-mentioned by a cron tick and their publishedAt is back-filled. But the when:1d ingest gate (PR #3417) blocks re-ingestion of months-old articles, so those entries will never receive a re-mention. The only automated cleanup path is natural TTL expiry (~7 days).
Consider documenting this limitation explicitly in the runbook, and/or switching step 3 to --mode=url for the immediate residue (which at least catches any direct-URL institutional entries), with a note that Google-News-routed residue must drain via TTL.
| // briefings-statements covers daily press releases; presidential-actions | ||
| // covers EOs / proclamations / nominations. | ||
| { name: 'White House', url: 'https://www.whitehouse.gov/briefings-statements/feed/' }, | ||
| { name: 'White House Actions', url: 'https://www.whitehouse.gov/presidential-actions/feed/' }, |
There was a problem hiding this comment.
White House Actions items may always be filtered by the 48 h ingest gate
The PR's own verification snapshot shows all 5 items in the White House Actions feed are 107–129 h old (≈ 4.5–5.4 days). The existing ingest-time freshness gate in list-feed-digest.ts (resolveMaxAgeMs(), default NEWS_MAX_AGE_HOURS=48) drops every item whose publishedAt < Date.now() - 48h. All 5 verified items would be dropped at ingest.
Presidential actions (EOs, proclamations, nominations) are signed irregularly — often spaced days apart. In low-activity periods the feed will routinely contain nothing newer than 48 h, so the source contributes zero items and the tier-1 boost is never triggered, defeating the purpose of adding "White House Actions": 1 to source-tiers.json.
Consider whether NEWS_MAX_AGE_HOURS should be raised for this source (or whether a per-feed max-age override is needed), or document the expected low-activity behaviour so operators aren't surprised when this feed stays silent.
| const readMaxAgeHoursRaw = Number.parseInt(process.env.DIGEST_READ_MAX_AGE_HOURS ?? '', 10); | ||
| const readMaxAgeHours = Number.isInteger(readMaxAgeHoursRaw) && readMaxAgeHoursRaw > 0 | ||
| ? readMaxAgeHoursRaw | ||
| : 48; | ||
| const readMaxAgeMs = readMaxAgeHours * 60 * 60 * 1000; | ||
| const nowMs = Date.now(); |
There was a problem hiding this comment.
readMaxAgeHours / nowMs re-computed on every buildDigest call
buildDigest is invoked once per rule in a loop. Parsing process.env.DIGEST_READ_MAX_AGE_HOURS and calling Date.now() on every invocation is harmless for correctness but means the effective cutoff can drift slightly between rules processed in the same tick. Lifting both constants out of the function (or computing them once in the caller) would make the tick-level freshness boundary consistent across all rules.
…e (P1×2 + P2/P3) 5 fixes responding to review: P1 (reviewer): READ-time floor + audit --mode=age both fall through on missing publishedAt — the legacy state of EVERY pre-PR-3422 row. So the actual residue this PR was meant to evict (Pentagon items in brief 2026-04-26-0802) wasn't catchable by either gate. Add --mode=residue that matches rows with missing/unparseable publishedAt — the explicit opt-in cleanup signal for the post-deploy one-shot. Operator runbook updated: wait at least 1 cron cycle (so still-active rows have publishedAt re-mentioned in via HSET), then run --mode=residue --apply. P1 (self-review): READ-time floor was hardcoded 48h, breaking weekly users (their 7d window stories all dropped). Replaced with readTimeAgeCutoffMs(windowStartMs) = windowStart - 24h buffer. Daily user (24h window) -> 48h cutoff (matches old behavior); weekly user (7d window) -> 8d cutoff (the broken case). DIGEST_READ_MAX_AGE_HOURS env var removed — there's no scenario where an operator wants to drop items younger than the rule's intended window. P2: Defensive cast in buildStoryTrackHsetFields. Number.isFinite guard on item.publishedAt prevents the literal "undefined"/"NaN" string from ever being persisted; degraded input writes '' which the read-side parseInt treats as legacy back-compat (fall through, not mis-classify as a stale-row). P2: Extract shouldDropTrackByAge + readTimeAgeCutoffMs into scripts/lib/digest-orchestration-helpers.mjs (the existing module for this purpose). 6 new unit tests cover the predicate matrix (within/at-cutoff/before/missing/unparseable/zero/null-track) plus 2 integration cases that reproduce the exact failure modes: - weekly user with 5d-old story is KEPT (window-aware vs naive 48h) - 4-month-old residue (Jan 2026 Pentagon PDF) is DROPPED for daily user P3: Audit parseArgs now rejects unknown args (--mode age space-not-equals) with exit-2 + clear error message. classifyTrack refactored to a pure function (mode + maxAgeMs as options) so tests can exercise it without the script's module-load argv side effects. main() invocation gated behind import.meta.url === file://process.argv[1] check (standard ESM idiom). 10 new unit tests cover the url/age/residue/both modes + parseArgs flag handling. 316/316 tests pass; importance-score-parity preserved.
…#3422) P2 (reviewer): --mode=residue --apply was too broad. Classifier matched ANY row with missing publishedAt — but every pre-PR-3422 row lacks the field, so the recommended one-shot cleanup could delete legitimate recent stories that simply hadn't been re-mentioned in the first post-deploy cron cycle. Fix: residue mode now requires BOTH conditions: 1. publishedAt missing/unparseable (legacy or anomalous), AND 2. lastSeen >= residueMinStaleMs ago (default 24h). Active stories get HSET-touched on every re-mention, so a row that hasn't been touched in 24h is genuinely abandoned — the new ingest gate (when:1d) is correctly excluding it. Fresh-lastSeen rows are PROTECTED from deletion. New flag: --residue-min-stale-hours=N (default 24, positive-int only). Operator runbook updated: 1. Wait >=24h post-deploy. 2. Dry run: node scripts/audit-static-page-contamination.mjs --mode=residue 3. Inspect output, confirm rows are stale + look like residue. 4. Apply: --mode=residue --apply Defensive fall-throughs: - Missing lastSeen: treat as ancient (errs toward eviction). Acceptable in this opt-in destructive mode; rows without lastSeen are anomalous. - publishedAt present (any value): NOT residue (caught by --mode=age). Tests: - 8 residue-mode cases incl. the explicit P2 fix ("fresh lastSeen must protect the row"), boundary at exactly the threshold, missing-lastSeen treated as ancient. - 4 parseArgs cases for the new --residue-min-stale-hours flag. P3 (reviewer): tests/digest-orchestration-helpers.test.mjs had a trailing blank line at EOF. Stripped. 323/323 tests pass; importance-score-parity preserved.
…ound-2) Reviewer flagged that --mode=residue --apply with the 24h staleness default would delete legitimate weekly-user stories. A pre-PR-3422 row that lacks publishedAt with lastSeen 2-7d ago is still ship-able for a weekly digest (whose readTimeAgeCutoffMs in digest-orchestration-helpers.mjs is windowStart - 24h = 8d ago) — but the 24h gate would mark it as residue and delete it. Fix: default --residue-min-stale-hours from 24 to 192 (= 7d max digest window + 24h buffer). Aligns with the readTimeAgeCutoffMs formula so residue mode can never delete a row still legitimately ship-able for ANY user. New explicit regression test: "SAFETY: does NOT match weekly-user 5d-old story" — pre-fix would have failed (deleted the row); now PROTECTED. Operators with confidence the fleet is daily-only can drop to faster cleanup via --residue-min-stale-hours=48 (documented escape hatch in the script header runbook + test case). The other safety guards remain: - publishedAt present → not residue (caught by --mode=age, not residue). - lastSeen fresh (< gate) → protected. - missing lastSeen → ancient (errs toward eviction in opt-in path). 325/325 tests pass; importance-score-parity preserved.
Why this matters
Brief 2026-04-26-0802 surfaced two months-old Pentagon items to a paying user despite PR #3417's
when:1dingest gate working perfectly (verified live: the gated query returns zero Pentagon items in the last 24h). The cause was post-deploy residue: pre-PR-3417 cron ticks wrote those items to story:track:v1 + the digest accumulator with their real Jan/Sep pubDates, andbuildDigest's read path had nopublishedAt-based freshness check. Pre-deploy entries stayed visible for up to ~48h until accumulator TTL drainage.This PR closes the residue gap structurally + reduces dependence on Google News for the gov sources where direct RSS exists.
What changed
1. READ-time freshness floor in
buildDigest(scripts/seed-digest-notifications.mjs)Drops
story:track:v1rows whosepublishedAtis older thanDIGEST_READ_MAX_AGE_HOURS(default 48h). Closes the residue gap for any future ingest-side change. Rows missingpublishedAt(legacy entries) fall through — back-compat. Operator kill switch:DIGEST_READ_MAX_AGE_HOURS=999.2. Persist
publishedAtinstory:track:v1HSET (server/worldmonitor/news/v1/list-feed-digest.ts)Adds
'publishedAt', String(item.publishedAt)tobuildStoryTrackHsetFields. Required for #1 AND for U6's age-mode (#3). Pre-existing rows pick it up on next mention.3. U6 audit gains
--mode=age|url|both(scripts/audit-static-page-contamination.mjs)Original URL classifier was BLIND to Google-News-routed entries (
track.linkis anews.google.com/rss/articles/CBMi…opaque redirect).--mode=agematches bytrack.publishedAt > max-age-hours, the right primary signal for evicting post-deploy residue. Per-reason rollup, operator runbook in script header.4. Feed swap: Pentagon + White House → direct RSS (
server/worldmonitor/news/v1/_feeds.ts)Verified via live
curl+ end-to-endparseRssXmlsmoke test:https://www.war.gov/DesktopModules/ArticleCS/RSS.ashx?ContentType=1&Site=945https://www.whitehouse.gov/briefings-statements/feed/https://www.whitehouse.gov/presidential-actions/feed/Direct RSS = publisher's pubDate is authoritative. State Dept, Treasury, DOJ retain
gn(...)with explanatory comment — no working public RSS at any verified path; the READ-time freshness floor (#1) mitigates the residue gap for those.5.
source-tiers.jsonaddsWhite House Actions: 1(+ scripts/shared mirror)Without this the new feed would default to tier-4 (no source boost).
Tests
250/250 pass across the touched + parity-coupled surface. New test asserts
publishedAtis emitted as a numeric string round-tripping throughparseInt.Operator runbook for the immediate residue
publishedAton existing entries via HSET re-mention.node scripts/audit-static-page-contamination.mjs --mode=agenode scripts/audit-static-page-contamination.mjs --mode=age --applyTest plan
[digest] buildDigest read-time freshness floor dropped N stale itemslines (= the READ-time gate is firing on residue)story:track:v1:*rows updated by recent cron ticks contain apublishedAtfield--mode=agedry-run captured in this PR's review thread before--applypublishedAtmore than 48h old