Skip to content

fix(brief-ingest): READ-time freshness + direct-RSS migration + U6 age mode#3422

Merged
koala73 merged 4 commits into
mainfrom
fix/brief-readtime-freshness-and-direct-rss
Apr 26, 2026
Merged

fix(brief-ingest): READ-time freshness + direct-RSS migration + U6 age mode#3422
koala73 merged 4 commits into
mainfrom
fix/brief-readtime-freshness-and-direct-rss

Conversation

@koala73

@koala73 koala73 commented Apr 26, 2026

Copy link
Copy Markdown
Owner

Why this matters

Brief 2026-04-26-0802 surfaced two months-old Pentagon items to a paying user despite PR #3417's when:1d ingest gate working perfectly (verified live: the gated query returns zero Pentagon items in the last 24h). The cause was post-deploy residue: pre-PR-3417 cron ticks wrote those items to story:track:v1 + the digest accumulator with their real Jan/Sep pubDates, and buildDigest's read path had no publishedAt-based freshness check. Pre-deploy entries stayed visible for up to ~48h until accumulator TTL drainage.

This PR closes the residue gap structurally + reduces dependence on Google News for the gov sources where direct RSS exists.

What changed

1. READ-time freshness floor in buildDigest (scripts/seed-digest-notifications.mjs)

Drops story:track:v1 rows whose publishedAt is older than DIGEST_READ_MAX_AGE_HOURS (default 48h). Closes the residue gap for any future ingest-side change. Rows missing publishedAt (legacy entries) fall through — back-compat. Operator kill switch: DIGEST_READ_MAX_AGE_HOURS=999.

2. Persist publishedAt in story:track:v1 HSET (server/worldmonitor/news/v1/list-feed-digest.ts)

Adds 'publishedAt', String(item.publishedAt) to buildStoryTrackHsetFields. Required for #1 AND for U6's age-mode (#3). Pre-existing rows pick it up on next mention.

3. U6 audit gains --mode=age|url|both (scripts/audit-static-page-contamination.mjs)

Original URL classifier was BLIND to Google-News-routed entries (track.link is a news.google.com/rss/articles/CBMi… opaque redirect). --mode=age matches by track.publishedAt > max-age-hours, the right primary signal for evicting post-deploy residue. Per-reason rollup, operator runbook in script header.

4. Feed swap: Pentagon + White House → direct RSS (server/worldmonitor/news/v1/_feeds.ts)

Verified via live curl + end-to-end parseRssXml smoke test:

Feed URL Verified
Pentagon (war.gov) https://www.war.gov/DesktopModules/ArticleCS/RSS.ashx?ContentType=1&Site=945 5/5 items, ages 33-54h
White House (briefings) https://www.whitehouse.gov/briefings-statements/feed/ 5/5 items, ages 13-59h
White House Actions (NEW) https://www.whitehouse.gov/presidential-actions/feed/ 5/5 items, ages 107-129h

Direct RSS = publisher's pubDate is authoritative. State Dept, Treasury, DOJ retain gn(...) with explanatory comment — no working public RSS at any verified path; the READ-time freshness floor (#1) mitigates the residue gap for those.

5. source-tiers.json adds White House Actions: 1 (+ scripts/shared mirror)

Without this the new feed would default to tier-4 (no source boost).

Tests

250/250 pass across the touched + parity-coupled surface. New test asserts publishedAt is emitted as a numeric string round-tripping through parseInt.

Operator runbook for the immediate residue

  1. Merge this PR → Vercel deploy + Railway redeploy.
  2. Wait ~1h for cron tick to start writing publishedAt on existing entries via HSET re-mention.
  3. Dry run: node scripts/audit-static-page-contamination.mjs --mode=age
  4. Inspect output — matched titles should look stale (months-old).
  5. Apply: node scripts/audit-static-page-contamination.mjs --mode=age --apply
  6. Random spot-check 3 user briefs over the next 24h — confirm no months-old items.

Test plan

  • CI green
  • After deploy: Railway logs show [digest] buildDigest read-time freshness floor dropped N stale items lines (= the READ-time gate is firing on residue)
  • After deploy: story:track:v1:* rows updated by recent cron ticks contain a publishedAt field
  • U6 audit --mode=age dry-run captured in this PR's review thread before --apply
  • 24h post-deploy + audit-apply: random spot-check on 3 user briefs — no items with publishedAt more than 48h old

…e mode

Three coupled gap closures discovered after PR #3419 merged. Brief
2026-04-26-0802 still surfaced two months-old Pentagon items DESPITE
PR #3417's `when:1d` ingest gate working perfectly (verified live: the
gated query returns zero Pentagon items in the last 24h). The cause was
post-deploy residue, not the gate. See:
  skill: ingest-gate-tightening-leaves-residue-in-read-path

Fixes (all on the same PR because they share the publishedAt persistence
precondition):

1. READ-time freshness floor in buildDigest (seed-digest-notifications.mjs).
   Drops story:track:v1 rows whose source publishedAt is older than
   DIGEST_READ_MAX_AGE_HOURS (default 48h). Closes the residue window: when
   any future ingest-side gate is tightened, pre-deploy entries stop
   shipping in briefs even before they TTL-expire from the accumulator.
   Rows missing publishedAt (legacy pre-this-PR entries) fall through —
   back-compat. Operator kill switch: DIGEST_READ_MAX_AGE_HOURS=999.

2. Persist publishedAt in story:track:v1 HSET (list-feed-digest.ts).
   Adds 'publishedAt', String(item.publishedAt) to buildStoryTrackHsetFields.
   Required for #1 AND for U6's age-mode (#3) to actually find the field.
   Pre-existing rows pick it up on their next mention.
   Test: tests/news-story-track-description-persistence.test.mts asserts
   the field is emitted as a numeric string that round-trips through
   parseInt.

3. U6 audit gains --mode=age|url|both (audit-static-page-contamination.mjs).
   Original URL-pattern classifier is BLIND to Google-News-routed entries
   (track.link is news.google.com/rss/articles/CBMi opaque redirect).
   --mode=age matches by track.publishedAt > max-age-hours, the right
   primary signal for evicting post-deploy residue. Per-reason rollup
   surfaces url vs age hits. Operator runbook included in the script
   header. Default --mode=url is back-compat.

4. Feed swap: 3 of 5 gov sources to direct first-party RSS (_feeds.ts).
   Verified 2026-04-26 via live curl + end-to-end parseRssXml smoke test:
     - Pentagon -> https://www.war.gov/DesktopModules/ArticleCS/RSS.ashx?ContentType=1&Site=945
     - White House (briefings) -> /briefings-statements/feed/
     - White House Actions    -> /presidential-actions/feed/ (NEW entry)
   Direct RSS means the publisher's pubDate is authoritative — no
   re-indexing of months-old PDFs creating fresh-looking Google entries.
   State Dept, Treasury, DOJ retain gn(...) with an explanatory comment;
   no working public RSS at any verified path, Federal Register fallback
   bot-blocked. The READ-time freshness floor (#1) mitigates the residue
   gap for those.

5. source-tiers.json: White House Actions = 1 (with scripts/shared mirror).
   New feed gets the same tier-1 source-boost as the existing White House
   entry; without it, Actions items would default to tier-4 with no boost.

250/250 tests pass; importance-score-parity preserved.

Operator runbook for the immediate post-deploy residue:
  1. Wait ~1h for cron tick to start writing publishedAt on existing
     entries via HSET re-mention.
  2. Dry run: node scripts/audit-static-page-contamination.mjs --mode=age
  3. Inspect output, confirm matched titles look stale.
  4. Apply: --mode=age --apply
@vercel

vercel Bot commented Apr 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview Apr 26, 2026 6:30am

Request Review

@greptile-apps

greptile-apps Bot commented Apr 26, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a READ-time publishedAt freshness filter to buildDigest, persists publishedAt in story:track:v1 HSET on every mention, swaps Pentagon and White House feeds to direct RSS, extends the U6 audit script with --mode=age|url|both, and adds tier-1 credit for "White House Actions".

  • Runbook gap (P1): The operator runbook for "immediate residue" directs --mode=age, but pre-deploy story:track:v1 entries have no publishedAt field (this PR is the first to persist it). classifyTrack's age branch evaluates Number.isInteger(NaN) → false and produces zero matches, giving false confidence the residue was evicted. The when:1d ingest gate (PR fix(news-ingest): drop static institutional pages before they reach the brief (PR 1/4) #3417) also prevents re-ingestion of months-old articles, so those entries will never receive a back-fill re-mention and must drain via 7-day TTL instead.
  • Silent feed (P2): The White House Actions feed had all 5 verified items at 107–129 h old — all above the 48 h NEWS_MAX_AGE_HOURS ingest gate — so the feed will silently contribute nothing during low-activity periods.

Confidence Score: 3/5

Safe to merge for the structural fixes; the operator runbook misleads on residue cleanup scope, and the new White House Actions feed may silently produce no content

One P1 (runbook --mode=age is a no-op for immediate pre-deploy residue, creating false cleanup confidence) and one P2 (White House Actions items exceed the ingest freshness gate at verification time). P1 ceiling is 4/5; the practical impact — stale items continuing to surface until TTL drains rather than being cleanly evicted — pulls the score to 3.

scripts/audit-static-page-contamination.mjs (age-mode logic vs. legacy entries) and server/worldmonitor/news/v1/_feeds.ts (White House Actions freshness window)

Important Files Changed

Filename Overview
scripts/audit-static-page-contamination.mjs Adds --mode=age
scripts/seed-digest-notifications.mjs Adds READ-time freshness filter in buildDigest; correct back-compat for legacy entries (no publishedAt = pass-through), env-var parsing duplicated per call
server/worldmonitor/news/v1/_feeds.ts Swaps Pentagon + White House to direct RSS; White House Actions added but all verified items (107-129h) exceed the 48h ingest freshness gate
server/worldmonitor/news/v1/list-feed-digest.ts Adds publishedAt to buildStoryTrackHsetFields; correct — field is always a valid number by this point (undated items are dropped upstream)
shared/source-tiers.json Adds White House Actions at tier 1; mirrors scripts/shared/source-tiers.json correctly
scripts/shared/source-tiers.json Mirror of shared/source-tiers.json — White House Actions tier 1 added consistently
tests/news-story-track-description-persistence.test.mts New test asserts publishedAt is emitted as a numeric string and round-trips through parseInt; correctly captures the READ-time gate contract

Sequence Diagram

sequenceDiagram
    participant RSS as Direct RSS / gn()
    participant Ingest as list-feed-digest<br/>(ingest-time gate)
    participant Redis as story:track:v1<br/>(Redis HSET)
    participant Digest as seed-digest-notifications<br/>(READ-time gate)
    participant User as User Brief

    RSS->>Ingest: raw items (with pubDate)
    Ingest->>Ingest: resolveMaxAgeMs() — drop if publishedAt > 48h
    Ingest->>Redis: HSET publishedAt=String(item.publishedAt)
    Redis->>Digest: HGETALL story:track:v1:*
    Digest->>Digest: READ-time gate — drop if publishedAt present AND > 48h
    note over Digest: legacy entries (no publishedAt) fall through
    Digest->>User: filtered digest stories
    note over Redis,Digest: Pre-deploy entries: no publishedAt → both READ-time gate and mode=age audit are blind
Loading

Reviews (1): Last reviewed commit: "fix(brief-ingest): READ-time freshness +..." | Re-trigger Greptile

Comment on lines 178 to 206
return { byHost: fmt(hostCounts), byPathPattern: fmt(pathPatternCounts) };
}

function classifyTrack(track, nowMs) {
const reasons = [];
const link = typeof track.link === 'string' ? track.link : '';

if ((MODE === 'url' || MODE === 'both') && link && isInstitutionalStaticPage(link)) {
reasons.push('url');
}

if (MODE === 'age' || MODE === 'both') {
const pubMs = Number.parseInt(track.publishedAt ?? '', 10);
if (Number.isInteger(pubMs) && pubMs > 0 && nowMs - pubMs > MAX_AGE_MS) {
reasons.push('age');
}
}

return reasons;
}

async function main() {
const { url, token } = getRedisCredentials();
console.log(`[audit] mode=${APPLY ? 'APPLY (will DELETE)' : 'DRY RUN'}`);
console.log(
`[audit] mode=${MODE} ${APPLY ? 'APPLY (will DELETE)' : 'DRY RUN'}` +
(MODE === 'age' || MODE === 'both' ? ` max-age-hours=${ARGS.maxAgeHours}` : ''),
);
console.log(`[audit] scanning ${SCAN_PATTERN}…`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 --mode=age is a no-op for the immediate pre-deploy residue

The operator runbook (PR description, steps 3-4) directs operators to run --mode=age for the "immediate residue" — the months-old Pentagon/gov entries already in Redis. But those entries were written before this PR, which is the first commit that persists publishedAt to story:track:v1 HSET fields. They have no publishedAt key.

In classifyTrack, the age branch is:

const pubMs = Number.parseInt(track.publishedAt ?? '', 10);
if (Number.isInteger(pubMs) && pubMs > 0 && nowMs - pubMs > MAX_AGE_MS) {
  reasons.push('age');
}

Number.parseInt("", 10)NaN; Number.isInteger(NaN)false → no match. The same back-compat fallthrough that protects the buildDigest filter also makes the age-mode audit blind to every entry written before this deploy.

The runbook's step 4 — "matched titles should look stale (months-old)" — will see zero matches, giving false confidence that the residue has been evicted.

Pre-deploy stale Google-News-routed entries (the exact ones --mode=age was designed for) cannot be evicted by this tool until they are re-mentioned by a cron tick and their publishedAt is back-filled. But the when:1d ingest gate (PR #3417) blocks re-ingestion of months-old articles, so those entries will never receive a re-mention. The only automated cleanup path is natural TTL expiry (~7 days).

Consider documenting this limitation explicitly in the runbook, and/or switching step 3 to --mode=url for the immediate residue (which at least catches any direct-URL institutional entries), with a note that Google-News-routed residue must drain via TTL.

Comment on lines +73 to +76
// briefings-statements covers daily press releases; presidential-actions
// covers EOs / proclamations / nominations.
{ name: 'White House', url: 'https://www.whitehouse.gov/briefings-statements/feed/' },
{ name: 'White House Actions', url: 'https://www.whitehouse.gov/presidential-actions/feed/' },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 White House Actions items may always be filtered by the 48 h ingest gate

The PR's own verification snapshot shows all 5 items in the White House Actions feed are 107–129 h old (≈ 4.5–5.4 days). The existing ingest-time freshness gate in list-feed-digest.ts (resolveMaxAgeMs(), default NEWS_MAX_AGE_HOURS=48) drops every item whose publishedAt < Date.now() - 48h. All 5 verified items would be dropped at ingest.

Presidential actions (EOs, proclamations, nominations) are signed irregularly — often spaced days apart. In low-activity periods the feed will routinely contain nothing newer than 48 h, so the source contributes zero items and the tier-1 boost is never triggered, defeating the purpose of adding "White House Actions": 1 to source-tiers.json.

Consider whether NEWS_MAX_AGE_HOURS should be raised for this source (or whether a per-feed max-age override is needed), or document the expected low-activity behaviour so operators aren't surprised when this feed stays silent.

Comment thread scripts/seed-digest-notifications.mjs Outdated
Comment on lines +466 to +471
const readMaxAgeHoursRaw = Number.parseInt(process.env.DIGEST_READ_MAX_AGE_HOURS ?? '', 10);
const readMaxAgeHours = Number.isInteger(readMaxAgeHoursRaw) && readMaxAgeHoursRaw > 0
? readMaxAgeHoursRaw
: 48;
const readMaxAgeMs = readMaxAgeHours * 60 * 60 * 1000;
const nowMs = Date.now();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 readMaxAgeHours / nowMs re-computed on every buildDigest call

buildDigest is invoked once per rule in a loop. Parsing process.env.DIGEST_READ_MAX_AGE_HOURS and calling Date.now() on every invocation is harmless for correctness but means the effective cutoff can drift slightly between rules processed in the same tick. Lifting both constants out of the function (or computing them once in the caller) would make the tick-level freshness boundary consistent across all rules.

…e (P1×2 + P2/P3)

5 fixes responding to review:

P1 (reviewer): READ-time floor + audit --mode=age both fall through on
missing publishedAt — the legacy state of EVERY pre-PR-3422 row. So
the actual residue this PR was meant to evict (Pentagon items in brief
2026-04-26-0802) wasn't catchable by either gate. Add --mode=residue
that matches rows with missing/unparseable publishedAt — the explicit
opt-in cleanup signal for the post-deploy one-shot. Operator runbook
updated: wait at least 1 cron cycle (so still-active rows have publishedAt
re-mentioned in via HSET), then run --mode=residue --apply.

P1 (self-review): READ-time floor was hardcoded 48h, breaking weekly
users (their 7d window stories all dropped). Replaced with
readTimeAgeCutoffMs(windowStartMs) = windowStart - 24h buffer. Daily
user (24h window) -> 48h cutoff (matches old behavior); weekly user
(7d window) -> 8d cutoff (the broken case). DIGEST_READ_MAX_AGE_HOURS
env var removed — there's no scenario where an operator wants to drop
items younger than the rule's intended window.

P2: Defensive cast in buildStoryTrackHsetFields. Number.isFinite guard
on item.publishedAt prevents the literal "undefined"/"NaN" string from
ever being persisted; degraded input writes '' which the read-side
parseInt treats as legacy back-compat (fall through, not mis-classify
as a stale-row).

P2: Extract shouldDropTrackByAge + readTimeAgeCutoffMs into
scripts/lib/digest-orchestration-helpers.mjs (the existing module for
this purpose). 6 new unit tests cover the predicate matrix
(within/at-cutoff/before/missing/unparseable/zero/null-track) plus 2
integration cases that reproduce the exact failure modes:
  - weekly user with 5d-old story is KEPT (window-aware vs naive 48h)
  - 4-month-old residue (Jan 2026 Pentagon PDF) is DROPPED for daily user

P3: Audit parseArgs now rejects unknown args (--mode age space-not-equals)
with exit-2 + clear error message. classifyTrack refactored to a pure
function (mode + maxAgeMs as options) so tests can exercise it without
the script's module-load argv side effects. main() invocation gated
behind import.meta.url === file://process.argv[1] check (standard
ESM idiom). 10 new unit tests cover the url/age/residue/both modes
+ parseArgs flag handling.

316/316 tests pass; importance-score-parity preserved.
…#3422)

P2 (reviewer): --mode=residue --apply was too broad. Classifier matched
ANY row with missing publishedAt — but every pre-PR-3422 row lacks the
field, so the recommended one-shot cleanup could delete legitimate recent
stories that simply hadn't been re-mentioned in the first post-deploy
cron cycle.

Fix: residue mode now requires BOTH conditions:
  1. publishedAt missing/unparseable (legacy or anomalous), AND
  2. lastSeen >= residueMinStaleMs ago (default 24h).

Active stories get HSET-touched on every re-mention, so a row that
hasn't been touched in 24h is genuinely abandoned — the new ingest gate
(when:1d) is correctly excluding it. Fresh-lastSeen rows are PROTECTED
from deletion.

New flag: --residue-min-stale-hours=N (default 24, positive-int only).
Operator runbook updated:
  1. Wait >=24h post-deploy.
  2. Dry run: node scripts/audit-static-page-contamination.mjs --mode=residue
  3. Inspect output, confirm rows are stale + look like residue.
  4. Apply: --mode=residue --apply

Defensive fall-throughs:
  - Missing lastSeen: treat as ancient (errs toward eviction). Acceptable
    in this opt-in destructive mode; rows without lastSeen are anomalous.
  - publishedAt present (any value): NOT residue (caught by --mode=age).

Tests:
  - 8 residue-mode cases incl. the explicit P2 fix ("fresh lastSeen must
    protect the row"), boundary at exactly the threshold, missing-lastSeen
    treated as ancient.
  - 4 parseArgs cases for the new --residue-min-stale-hours flag.

P3 (reviewer): tests/digest-orchestration-helpers.test.mjs had a trailing
blank line at EOF. Stripped.

323/323 tests pass; importance-score-parity preserved.
…ound-2)

Reviewer flagged that --mode=residue --apply with the 24h staleness
default would delete legitimate weekly-user stories. A pre-PR-3422 row
that lacks publishedAt with lastSeen 2-7d ago is still ship-able for a
weekly digest (whose readTimeAgeCutoffMs in
digest-orchestration-helpers.mjs is windowStart - 24h = 8d ago) — but
the 24h gate would mark it as residue and delete it.

Fix: default --residue-min-stale-hours from 24 to 192 (= 7d max digest
window + 24h buffer). Aligns with the readTimeAgeCutoffMs formula so
residue mode can never delete a row still legitimately ship-able for
ANY user. New explicit regression test:
  "SAFETY: does NOT match weekly-user 5d-old story" — pre-fix would
  have failed (deleted the row); now PROTECTED.

Operators with confidence the fleet is daily-only can drop to faster
cleanup via --residue-min-stale-hours=48 (documented escape hatch in
the script header runbook + test case).

The other safety guards remain:
  - publishedAt present → not residue (caught by --mode=age, not residue).
  - lastSeen fresh (< gate) → protected.
  - missing lastSeen → ancient (errs toward eviction in opt-in path).

325/325 tests pass; importance-score-parity preserved.
@koala73
koala73 merged commit 6c28e93 into main Apr 26, 2026
10 checks passed
@koala73
koala73 deleted the fix/brief-readtime-freshness-and-direct-rss branch April 26, 2026 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant